Hands reached to write another addition to my short course in computer graphics. So, the topic for the next conversation is the use of normal maps . What is the main difference between using normal maps and shading phong? The main difference in the density of job information. To shade Phong, we used normal vectors given to each vertex of our polygonal mesh, interpolating the normals inside the triangles. Using normal maps allows you to set normals for each point on our surface, and not just occasionally, which simply dramatically affects the detail of images.

In principle, in the lecture about shaders, we already used the normal map, but only given in the global coordinate system. Now the conversation will be abouttangent space . So, here are two textures, the left one is set in global space (RGB is directly converted to the XYZ vector), and the right one is tangent.


To use the normal specified in the tangent space, for the pixel to be drawn, we compute the tangent frame (Darboux frame). In this frame, one vector (z) is orthogonal to the surface, and the other two define the tangent plane to the surface. Then we read the normal vector from the texture, and convert its coordinates from the newly calculated frame to the global one. Since the normal map most often defines only a small perturbation of the normal, the dominant color of the texture is blue.

It would seem, why such difficulties? Why not use a simple global benchmark as before? Imagine that we want to animate our model. For example, I took our old friend Negro and opened his mouth. Obviously, the modified surface must also have other normals!



Here is the model on the left, in which the mouth is open, and the normal map (global) has not been changed. Look at the mucous membrane of the lower lip. The light hits right in the face, with a model with a closed mouth, the mucous membrane, of course, was not illuminated in any way. The mouth opened, but it was still not lit ... The right picture was calculated using the normal map defined in the tangent space.

So, if we have an animated model, then to define a normal map in global space, we need one texture for each frame of the animation. And since the tangent space naturally follows the surface, such a texture is enough for us alone!

Here is a second example:



These are the textures for the Diablo model. Note that only one hand is visible on the texture. And only one half of the tail. The artist used the same texture for the left and right hands, and the same texture for the left and right parts of the tail. (By the way, this is what prevented us from calculating ambient occlusion .) And this means that in the global coordinate system I can set normal vectors for either the left hand or the right. But not for two at once!

So, we finish with motivation and go directly to the calculations.

Starting Point, Phong Shading

So let's look at the starting point . The shader is very simple, it's Phong shading.

struct Shader : public IShader {
    mat<2,3,float> varying_uv;  // triangle uv coordinates, written by the vertex shader, read by the fragment shader
    mat<3,3,float> varying_nrm; // normal per vertex to be interpolated by FS
    virtual Vec4f vertex(int iface, int nthvert) {
        varying_uv.set_col(nthvert, model->uv(iface, nthvert));
        varying_nrm.set_col(nthvert, proj<3>((Projection*ModelView).invert_transpose()*embed<4>(model->normal(iface, nthvert), 0.f)));
        Vec4f gl_Vertex = Projection*ModelView*embed<4>(model->vert(iface, nthvert));
        varying_tri.set_col(nthvert, gl_Vertex);
        return gl_Vertex;
    }
    virtual bool fragment(Vec3f bar, TGAColor &color) {
        Vec3f bn = (varying_nrm*bar).normalize();
        Vec2f uv = varying_uv*bar;
        float diff = std::max(0.f, bn*light_dir);
        color = model->diffuse(uv)*diff;
        return false;
    }
};

Here is the result of the shader:



For ease of training and debugging, I will remove the skin texture and apply the simplest regular grid with horizontal red and vertical blue lines:



Let's see how our Fong shader works on the example of this picture:



So, for each vertex of the triangle we have it p coordinates, its texture coordinates uv and normal vectors to the vertices n. To draw each pixel, the rasterizer gives us the barycentric coordinates of the pixel (alpha, beta, gamma). This means that the current pixel has spatial coordinates p = alpha p0 + beta p1 + gamma p2. We interpolate the texture coordinates in exactly the same way, then we interpolate the normal vector too:



Note that the red and blue lines are the isolines u and v, respectively. So, for each point on our surface, we define a (sliding) Darboux frame, in which the x axis is parallel to the red lines, the y axis is parallel to the blue, and z is orthogonal to the surface. It is in this benchmark that normal vectors are defined.

We calculate a linear function at its three points

So, our task is to calculate the triple of vectors that defines the Darbou rapper for each pixel being drawn. To get started, let's digress and imagine that a linear function f is given in our space that associates with each point (x, y, z) a real number f (x, y, z) = Ax + By + Cz + D. The only thing that we we don’t know the numbers (A, B, C, D), but we know the value of the function at the vertices of some triangle (p0, p1, p2):





We can imagine that f is just the height of some inclined plane. We fix three different points on the plane and we know the heights at these points. The red lines on the triangle show the contours of the height: the contour for height f0, for height f0 + 1 meter, f0 + 2 meters, etc. For a linear function, all these contours are obviously parallel lines.

What we are interested in is not so much the direction of the isolines as the direction they are orthogonal to. If we move along a certain contour, then the height does not change (thanks, captain), if we deviate the direction slightly from the contour, then the height begins to change, and the biggest change per unit of height will be when we move in the direction orthogonal to contours.

We recall that the direction of the steepest rise for a function is nothing but its gradient. For a linear function f (x, y, z) = Ax + By + Cz + D, its gradient is a constant vector (A, B, C). It is logical that it is constant, since any point on the plane is inclined equally. I remind you that we do not know the numbers (A, B, C). We only know the value of our function at three different points. Can we restore A, B and C? Of course.

So, we have three points p0, p1, p2 and three values ​​of the function f0, f1, f2. We are interested in finding the vector (A, B, C) giving the direction of the fastest growth of the function f. For convenience, we will consider the function g (p), which is defined as g (p) = f (p) - f (p0):



Obviously, we just moved our inclined plane without changing its inclination, therefore, the direction of the fastest growth for the function g will coincide with the direction of the fastest growth of the function f.

Let's rewrite the definition of g:



Note that the superscript p ^ x is the x coordinate of p, not the power. That is, the function g is just the scalar product of the vector connecting the current point p with the point p0 and the vector (A, B, C). But we still do not know (A, B, C)! Not scary, we’ll find them now.

So what do we know? We know that if we go to p2 from the point p0, then the function g will be equal to f2-f0. In other words, the scalar product between the vectors p2-p0 and ABC is f2-f0. The same goes for dot (p1-p0, ABC) = f1-f0. That is, we are looking for a vector (ABC) that is simultaneously orthogonal to the normal to the triangle and has these two restrictions on scalar products:



We write the same thing in matrix form:



That is, we got the matrix equation Ax = b, which is easily solved:



Note that I used letter A in two senses, the meaning should be clear from the context. That is, our 3x3 matrix A, multiplied by an unknown vector x = (A, B, C), gives the vector b = (f1-f0, f2-f0, 0). The unknown vector is found by multiplying the matrix inverse to A by vector b.

Note that there is nothing in matrix A that depends on the function f! It contains only information about the geometry of the triangle.

We compute the Darbou rapper and apply the normal map (disturbance)

In total, the Darboux frame is a triple of vectors (i, j, n), where n is a normal vector, and i and j can be calculated as follows:



Here is the final code using normal maps defined in tangent space, and here you can find changes in the code compared to Phong tinted.

Everything is pretty straightforward there, I calculate the matrix A:

        mat<3,3,float> A;
        A[0] = ndc_tri.col(1) - ndc_tri.col(0);
        A[1] = ndc_tri.col(2) - ndc_tri.col(0);
        A[2] = bn;

Then I compute Darbou's rapper vectors:

        mat<3,3,float> AI = A.invert();
        Vec3f i = AI * Vec3f(varying_uv[0][1] - varying_uv[0][0], varying_uv[0][2] - varying_uv[0][0], 0);
        Vec3f j = AI * Vec3f(varying_uv[1][1] - varying_uv[1][0], varying_uv[1][2] - varying_uv[1][0], 0);

Well, once they’ve calculated them, I read the normal from the texture, and I do the simplest transformation of coordinates from the Darbu frame to the global frame.
If anything, then the coordinate transformation I have already described .

Here is the final render, compare the detail with the Phong tint :

Debugging tip

It's time to remember how the lines are drawn . Add a regular red-blue grid to the model and for all the vertices draw the resulting vectors (i, j), they should coincide well with the direction of the red-blue lines of the texture.

Happy coding!