Why transform normals with the transpose of the inverse of the modelview matrix?

I am working on some shaders, and I need to transform normals.

I read in few tutorials the way you transform normals is you multiply them with the transpose of the inverse of the modelview matrix. But I can't find explanation of why is that so, and what is the logic behind that?

It flows from the definition of a normal.

Suppose you have the normal, N, and a vector, V, a tangent vector at the same position on the object as the normal. Then by definition N·V = 0.

Tangent vectors run in the same direction as the surface of an object. So if your surface is planar then the tangent is the difference between two identifiable points on the object. So if V = Q - R where Q and R are points on the surface then if you transform the object by B:

V' = BQ - BR
   = B(Q - R)
   = BV

The same logic applies for non-planar surfaces by considering limits.

In this case suppose you intend to transform the model by the matrix B. So B will be applied to the geometry. Then to figure out what to do to the normals you need to solve for the matrix, A so that:

(AN)·(BV) = 0

Turning that into a row versus column thing to eliminate the explicit dot product:

[tranpose(AN)](BV) = 0

Pull the transpose outside, eliminate the brackets:

transpose(N)*transpose(A)*B*V = 0

So that's "the transpose of the normal" [product with] "the transpose of the known transformation matrix" [product with] "the transformation we're solving for" [product with] "the vector on the surface of the model" = 0

But we started by stating that transpose(N)*V = 0, since that's the same as saying that N·V = 0. So to satisfy our constraints we need the middle part of the expression — transpose(A)*B — to go away.

Hence we can conclude that:

 transpose(A)*B = identity
 => transpose(A) = identity*inverse(B)
 => transpose(A) = inverse(B)
 => A = transpose(inverse(B))

My favorite proof is below where N is the normal and V is a tangent vector. Since they are perpendicular their dot product is zero. M is any 3x3 invertible transformation (M^-1 * M = I). N' and V' are the vectors transformed by M.

To get some intuition, consider the shear transformation below.

Note that this does not apply to tangent vectors.

Take a look at this tutorial:

https://paroj.github.io/gltut/Illumination/Tut09%20Normal%20Transformation.html

You can imagine that when the surface of a sphere stretches (so the sphere is scaled along one axis or something similar) the normals of that surface will all 'bend' towards each other. It turns out you need to invert the scale applied to the normals to achieve this. This is the same as transforming with the Inverse Transpose Matrix. The link above shows how to derive the inverse transpose matrix from this.

Also note that when the scale is uniform, you can simply pass the original matrix as normal matrix. Imagine the same sphere being scaled uniformly along all axes, the surface will not stretch or bend, nor will the normals.

If the model matrix is made of translation, rotation and scale, you don't need to do inverse transpose to calculate normal matrix. Simply divide the normal by squared scale and multiply by model matrix and we are done. You can extend that to any matrix with perpendicular axes, just calculate squared scale for each axes of the matrix you are using instead.

I wrote the details in my blog: https://lxjk.github.io/2017/10/01/Stop-Using-Normal-Matrix.html

Don't understand why you just don't zero out the 4th element of the direction vector before multiplying with the model matrix. No inverse or transpose needed. Think of the direction vector as the difference between two points. Move the two points with the rest of the model - they are still in the same relative position to the model. Take the difference between the two points to get the new direction, and the 4th element, cancels out to zero. Lot cheaper.