Detailing, reflections and post effects in GTA V

From a translator:

This publication is a continuation of the material “How is a frame rendered in GTA V” . Now the author considers the issues of detail, lighting and post-processing of the frame. Enjoy reading.

Level of detail

If we are talking about the absolute advantages of Rockstar over its competitors, then the level of detail of the company's products is definitely beyond praise. Los Santos is a whole galaxy of all kinds of scenes of varying degrees of detail / polygonality, with all the data being broadcast in real time and this does not block the loading screen for a minute. It’s just breathtaking!

City Lights

All the lights that are visible in the distance are real. You can drive closer and see lights that emit light in the distance.

With this statement, Aaron Garbut, one of the founders and art directors of Rockstar North, made shortly before the release of the game on the PS3.

Is it so? Let's look at such a night scene:


Lights “To”
Caution! Traffic!


Lights “After”

It turns out that it is: each small light spot is a quad visualized in the form of a tiny 32x32 texture, like the one you can see below.


All of them are tightly grouped within the framework of the scene geometry, but these are still the tens of thousands of polygons that the GPU has to work with.


Lights


Blueprint of lights


Depth test (pass / fail)

And it's not just about static geometry: car headlights also move along the road - the image is updated in real time. Of course, at a great distance there is no point in completely visualizing the models of cars, just 2 headlights are enough to create the illusion of movement. But if you decide to drive closer to one of the light sources, the level of detail rises and we get a full-fledged image of a car

Low Poly Graphics Grids

Back to the frame that we analyzed earlier. Some fairly large areas are processed in just one draw call. So it was, in particular, in the case of rendering a hill in this image: A


hill in just one call to the rendering team

So what is this tiny hill in the distance?

In fact, Vinewood Hills is by no means small. This is a large area of ​​several square kilometers with dozens of houses and other buildings.

On the top of the hill are the Galileo Observatory, Sisyphus Theater, and Vinewood Lake. To create all these sights - you can take a closer look at them or just drive past - it took thousands of draw calls and countless polygons.

But in our case, this zone is far away, and therefore is represented by a low-poly version: just one draw call, involving some 2,500 triangles.


Vinewood Hills Low Poly Model

Rendering is done using a single graphics grid that processes diffuse texture data. Even if there is some mechanism for converting the graphic grid into a low-poly version, it is impossible to fully automate the process, and therefore I won’t be surprised if 3D artists from Rockstar have to spend more than one day to manually adjust everything.

Another example of similar rendering in a single draw call is the Little Seoul area, which has combined several blocks of the city.


Low poly model of Little Seoul

Having a low-poly model of the virtual world on hand, it is much easier to visualize the texture of reflections, which does not require either detail or high resolution. For games of the same level of detail, real-time processing of cubic environmental textures is often very expensive or completely impossible due to the variety of geometric shapes used.

Gaming Resources

Creating several versions of the virtual world with different levels of detail, as is done in GTA, is a difficult and very laborious task. But even, it would seem, having reached your goal, you still remain halfway: you can have gigabytes of models and textures on your hard drive, but they all cost nothing until you find a suitable way to load them into RAM or to the memory of the GPU.

GTA V provides a game resource in real time by downloading / downloading models and textures as you move around the map. Moreover, the ability to use this content in real time and at the same time enjoy the game without interruption is really impressive.

From a technical point of view, the most difficult thing was to fit the necessary information into the console’s memory so that it still worked as smoothly as we see [...]. We can translate and compile much more, which allows us to provide a higher level of detail than it was in GTA 4.

Aaron Garbut

Of course, this data transfer mechanism also has its limitations: for example, when you select a character, the camera jumps from one section of the map to another, in this case the system is overloaded and, logically, it takes 5 seconds to navigate new conditions and switch. But in GTA V, such a transition was beaten very effectively with the help of an approaching / retreating animation / translation window, and so the loading of the screen is not felt at all.

As a rule, when you drive a car, the speed is low enough so that the throughput of the data transfer system allows you to gradually load the available updates. Another thing is airplanes: by the standards of streaming systems, they move too fast, and therefore they had to significantly reduce their speed relative to indicators that we are used to in reality. In addition, the detail level of graphic grids corresponding to flight scenes is much lower than in the case of walking / driving, which also facilitates data transfer. Although, when a game resource requests fresh updates, sometimes you have to resort to the aforementioned trick. A special option “High-resolution data transmission during flight” has been added to the PC version settings.

Reflection

Since there was not much water in the scene that I analyzed earlier, let's focus on visualizing the pool or ocean using the following scene as an example:


Pool

As we have seen, the scene is rendered according to the standard scheme: a cubic texture of the environment is generated, basically, it is necessary to visualize objects that reflect the environment.

This is what the scene looks like before rendering water:


Before rendering water.

Visualization of the water surface is a separate story that has nothing to do with creating a cubic texture.

Reflection map

First, a “plane reflection" map is generated. And with a very low resolution, 240x120, and the process itself is similar to the process of generating a cubic texture, but this time only one buffer is created, in which there is a ladder and characters.



The scene is visualized upside down, and only then, after applying the symmetry, it will be possible to get the correct reflection.

Refraction map

Part of the image is extracted - areas with a water surface - allowing you to create a refraction map. Thanks to this, the refraction effect is subsequently imitated: a glow from under water.

At this stage, blue is added to the “water opacity” (the deeper the deeper the shade), as well as the caustics. The final version of the map is half the size of a similar buffer.


Base


Map of Refraction

Combination

To combine different buffers, a rectangular polygon is drawn, which acts as the surface of the water in the pool. Given the texture of the relief, the normals of the polygon are shifted pixel by pixel, which simulates a slight ripple.

For the ocean, the normals are not shifted, and the entire graphic grid of a single frame is drawn with its tops up, which simulates the movement of waves.

Given the normals of individual pixels, the pixel shader analyzes the performance of the reflection and refraction map at different points, the coordinates are calculated using Fresnel equations.


Reflection map, refraction map and relief map


Water “Before”

And, as a result, we get the following image:


Water “after”

The result is pleasantly pleasing. In combination with a successful ripple effect, the water looks very realistic.

Mirrors

Mirrors are visualized in the same way as water. In this situation, everything is even much simpler, because the mirrors only reflect light and here it is not necessary to take into account its refraction.


Mirror reflection

Unlike water, the surface of the mirror is perfectly flat and motionless, and therefore it is not so easy to hide the very low resolution of the object - texels are very visible. By increasing the reflection parameters in the settings, we get a higher resolution.

To create a reflection map, an additional stage of rendering the scene is required, but this is not an easy task. The engine does not allow to start this process if the mirror is not visible in the doorway or the player is too far from it (in this case, the mirror resembles a black square).

Headlights

Remember that in the environmental map that is generated at the beginning of each frame, there are no characters, no cars, but only the main buildings and landscape?

How, in this case, the headlights of the car in the next screenshot are reflected in the wet asphalt?


Headlight Reflections

In the frame that I analyzed in Part 1 , this was not obvious since there was a day. But, in fact, after combining all the buffers, one by one the headlights are drawn. For each bulb, the amount of light emitted into other graphic grids is calculated, including strong reflection from glossy surfaces, such as wet asphalt.


Headlights 0%


Headlights 50%


Headlights 80%


Headlights 100%

For each light source, its own graphic grid is drawn: initially it looks like a cellular octahedron, but subsequently the vertex shader modifies the grid so that it matches the shape of the light halo.

This graphic grid is not textured, it only helps to group the pixels inside the light halo so that you can apply a pixel shader. The shader will allow you to dynamically calculate the lighting depending on the depth of the pixel, its distance to the light sources, its normals and the presence of mirror / glossy properties.

Below you see a blueprint of the graphic grid used to calculate the lighting of the road with a lantern.


Before meshing


After meshing

With such a consideration, one big advantage of delayed shading as compared to direct shading becomes obvious: a large number of light sources can be visualized in the scene, practically without the help of a pixel shader, which processes only pixels illuminated by certain objects. With direct shading, it is necessary to calculate the effect of several light sources on a fragment of the frame at once, even if it is not lit or subsequently overlapped by another fragment.

And since you are still reading my review, I propose moving on to its final part on post-effects.

Post Processing Effects

Post-effects are applied after rendering the scene to improve the quality of details, make adjustments, create a new atmosphere ...

In Part 1, we saw how individual post-effects were superimposed, in particular, glow, smoothing and tonal compression. Today we’ll talk about other effects found in GTA V.

Glare and light streaks

Sometimes, when light passes through a real lens, the scattering and internal refraction of the rays will lead to their distortion.

In this article, I called “glare” several bright spots along one axis formed by a bright light source and the center of the screen. There are also “light strips” - rays from a light source. This kind of distortion is very common in films, and when they appear in the game, a certain “cinematic” effect is created.


Glare and light strips

As a rule, 2 methods of visualizing such effects are used:

• Based on the image: the brightest areas are highlighted, which are subsequently duplicated and distorted. This option is suitable for any number of bright light sources.
• sprite-based: adds textured sprites whose position is manually changed. Each light source has to be processed separately, but the artist is given more opportunities to change the shape, color and intensity of highlights and light streaks ...

GTA V employs both methods: a method that uses images allows you to add a thin blue halo in the lower left corner of the frame. In fact, this is a symmetrical reflection of the contents of the brightness buffer. But the most noticeable distortions in this scene appear due to manipulations with sprites produced with the sun. First, light strips are added by visualizing 12 inverted quadrangles around the sun. Then 70 sprites are drawn along the axis “sun - screen center” for glare. The effects become more intense when you turn the camera in the sun.


Basis


Basis + streaks of light


Basis + streaks of light + glare


Basis + streaks of light + glare (ultra-sensitive blueprint)

The engine uses several sprites to simulate various lens distortion effects:


Distortion effects

And, as a result, we get the following result:


Basic image


Basic image + lens flare


Basic image + lens flare (blueprint)

In GTA V much attention is paid to details, and lens flare is no exception: their size is proportional to the aperture of the camera. So if you suddenly look at the sun, the glare will increase first, and then, as soon as the diaphragm narrows, it will decrease. The animation below illustrates this feature perfectly.

It is also pleasant that if you switch to the first-person mode, there will be practically no glare, because now you see everything with the eyes of a person, and not through the camera lens.


Wide aperture


Narrow aperture

Anamorphic lenses

At night or when playing on dark areas of the map, the effect of anamorphic lenses is modeled: long vertical or horizontal, usually blue, stripes. Recently, minor strips from anamorphic lenses have become widespread in the practice of Hollywood, because they often appear in new science fiction films.

Here, the effect is achieved using sprites, just like in the case of the sun's rays, which we examined earlier. True, it is used only on very bright, like car headlights, light sources directed directly into the camera.


Anamorphic effect "Before"


Anamorphic effect "After"

Depth of field

In the movie, the scene presented below would have looked a bit “artificial”: everything is too sharp and clear, but in the film the landscape in the background is out of focus, slightly blurred.


The base image

You just need to add depth of field (DoF), blurring areas of the image that remain out of focus.

How to do it? First, a blur map ( CoC ) is generated based on the depth buffer . It allows you to calculate how many pixels are out of focus, and, accordingly, sets the degree of forthcoming blur. The CoC value of an individual pixel depends solely on its distance to the camera (depth) and lens parameters.

However, it is worth noting that in GTA V CoC is assigned a value from -1 to 1, so you can find out whether the pixel is in front of the lens or behind the focus area. For example, pixels far beyond the focus are assigned CoC 1, and off-focus pixels that come very close to the camera are set to -1. Any indicator near 0 assumes that the pixel will be practically not blurred.

What are numerical values ​​for?

This is because achieving a decent depth of field is not so simple and you have to take into account many factors by applying a blur effect to the scene.

For example, you don’t want the pixel out of focus in the background to not overlap the adjacent one from the focus in front of it. But what if there is an inverse problem: you need the blurred foreground pixel to overlap a clear fragment behind it. As soon as it comes to blurring pixels, the question is not “how much blurring is”, you also have to analyze nearby fragments, which can be both in front and behind the studied pixel, both out and in focus ... The numerical format of CoC values ​​is noticeable facilitates the task.

On the CoC map below, I used the green channel for CoC> 0 and the red channel for CoC <0, which helps to orientate.


Depth


Map CoC Map

So, after studying the CoC map, it becomes clear that Leicester, highlighted in black, will be in focus, and Franklin in the foreground and the landscape in the background is out of focus.


CoC foreground blur map

Then the engine selects only “foreground areas out of focus”: all texels with CoC <0. This CoC foreground map is blurred using the computational shader, first horizontally and then vertically.

Why? Well, the CoC value for Franklin's pixels is approximately 0.7, while for the bench directly behind it, which came into focus, it is 0. Now that the blur is complete, Franklin’s image looks pretty blurry and the bench remains clear. moreover, the area next to Franklin's right hand looks strange: a sharp transition is obtained from a clearly blurred hand to extremely clear bench lines. Rigid borders are very noticeable ... Disorder, the contours should connect seamlessly, and Franklin's pixels should slightly overlap neighboring ones.

This is why this CoC blur card is needed: to smooth out the roughness of the CoC card and ensure that the foreground pixels are out of focus correctly.

Now we have everything we need to calculate the depth of field. Previously, the degree of blurring was calculated using a pixel shader at a lower resolution, separately horizontally and separately vertically, which guaranteed more accurate results. The GTA V team decided to keep the method with 2 phases, but chose to perform all the necessary operations with the original resolution, and to avoid performance degradation, use a computational shader instead of a pixel one.

And that’s smart, because computational shaders easily adapt to blur large areas. So when calculating the final value of the “blur” of a pixel, its color will depend on less or more neighboring ones (depending on the CoC of the pixel) and some of the latter will be excluded if there is a chance of incorrect compilation.


The base, depth, CoC card and CoC card of the blurred foreground

When using double blur using the computational shader, we get the following image:


Depth of field “Before”


Depth of field “After”

This effect gives the frame a completely different perception of the measurements, like in a movie when the director focuses a camera on the character to draw attention to him.

Conclusion

Of course, there are other post-processing effects that could be considered, but it seems that this series of articles on GTA V has turned out to be much larger than I planned.

There is imitation of haze, scattering of light in depth (sometimes bright areas overlap neighboring ones, sometimes appropriate graphic grids are manually added) or motion blur (like a mixed approach, when blurring is performed only in the direction of the camera’s movement, excluding player pixels due to the template buffer used in as a mask).

The “Wasted” screen, when your character dies, is also a pure post-effect: after the main rendering is completed, the scene is blurred, filled with gray, then a vignette is applied, graininess is added, as in a movie, and finally, a familiar text is drawn on top of all this.

I hope I still helped a little to understand how Rockstar was able to create a product that has become a true landmark in the world of video games. The endless universe, the specifics of the presentation and attention to detail, as well as the ability to run the game on the old-generation consoles made GTA V an amazing project.