The elusive problem of frame timing

Croteam Technical Director Alain Ladavach, who participated in the development of Serious Sam and Talos Principle, tells how he managed to find the reason for the inhibition of graphics even on the most powerful machines.

Finally, an explanation has appeared of why some games slow down on your PC (and a ray of hope that they will stop slowing down in the near future).

Tt-brakes

You were looking forward to the next part of your favorite PC video game series and it finally came out. This time you want to enjoy it in its entirety, so you spent money and time on thorough preparation. You replaced the processor, installed a state-of-the-art video card, added more RAM — damn it, even bought a RAID on the SSD. The game should be smooth with the screen saver.

Pre-order is finally unlocked and you have just completed the installation. In a nervous anticipation, you start the game for the first time. So far so good, it works at 60 frames per second. Or, at least, this is what the GPU's frame counter shows. But something is wrong. You make the mouse sharp, chaotic movement. Straight left-right, and then the game ... begins to slow down! Damn, how is this possible? How can she slow down at 60 frames per second?

If this has never happened to you, it may seem ridiculous. But if you have experienced them, then you will most likely hate brakes with all your heart. Brakes in games. This is not a good old lag. Not a low frame rate. These are just “brakes” occurring at high frame rates on ideal, super-fast machines. What is it, where did they come from and how to get rid of them? Let me tell you a story ...

Brakes, smoothness, speed ... are they the same thing?

Video games worked with a frequency of 60 fps since the days of the first arcade machines in the 70s. It is usually expected that the game works with the same frequency that is used by the display. This was the case before the popularization of 3D-games, in which for the first time reduced frame rates became permissible. In the 90s, when “ 3D maps ” (as we called them before they became “ GPUs ”) began to replace software rendering, people played games at 20 fps, and 35 fps was considered a decent value for serious network battles. I'm not joking .

Today we have super-fast cars and "of course, they can work at 60 fps"However, the number of users disappointed with the speed of games is greater than ever. How is this possible? The problem is not that games cannot work fast enough, but that they slow down even when they can work fast !

If you read different game forums, you will surely find similar messages:


You might think that this is a single problem, but look at the statistics of Google search queries:


Over the past five years, brakes (stutter) have become (relatively) more serious problem than speed!

(Note that these are relative values. They do not mean that, in general, people ask for braking more than about frame rates. They mean that requests for frame rate (frame rate) remain at the same level, and the number of requests for brakes is growing especially lately.)

Ten years of searching for the cause of unexplained brakes

The patient is more alive than dead, just slows down a little more than necessary.

I first encountered this problem around 2003. We worked on Serious Sam 2 , and users started sending us reports that they were testing something at an empty level, and the movements were not smooth as the mouse moved. This was accompanied by a very characteristic pattern on the frame rate graph, which we called the "cardiogram".

We thought that there was a bug somewhere in the code, but we could not find it. It seemed that the problem appeared and disappeared randomly - after the application was restarted, the machine was restarted ... and then the player changed some speed option and it disappeared. Then the player turned on this option again, but the problem did not return. She looked like a ghost.

Obviously, this problem arose not only in our country. Watching the same problems in other games, we began to think that the drivers were to blame. But this happened on video cards from different manufacturers. Even on different APIs (OpenGL, DirectX 9, DirectX 11 ...) - the only common thing they had was that they appeared on different machines, in some scenes ... sometimes.


Nessie, Bigfoot ... almost as elusive as the problem with the cardiogram.

We released a few more games, but this strange behavior still appeared and disappeared. It irritated some users, and we recommended them to change the speed options - sometimes it helped, sometimes not. Such is life, isn't it?

But one day, on a great winter day at the beginning of 2013, my colleague Dean called me to see another example of this problem, which he at that time could reproduce relatively stably. This time the problem arose at the level of Serious Sam 3 . We experimented with options in this scene, until it suddenly came to me. I understood what was the reason! And she was very simple - no wonder she escaped everyone for a decade.

By changing just one very simple game engine option, we were able to make this problem appear and disappear in this particular scene. But it immediately became obvious to us that its high-quality solution would require much more effort. Efforts not only from our side, but also the entire PC gaming ecosystem — programmers for GPU drivers, API developers, OS vendors — everyone.

Let me explain.

What was the reason all this time

I would like to show you on the example of a scene from Serious Sam 3, which Dean and I explored five years ago. Or even better - using the example of a test scene from Serious Sam 2, in which we first saw it. But, unfortunately, after replacing the “iron” this elusive beast can move to another scene. I have a scene from The Talos Principle , in which I recently managed to reproduce this problem, and I filmed several videos that made it possible to analyze it in more detail.

But before we start, make sure you actually watch the video at 60 fps. To view the examples below, switch to 1080p60, as shown in the picture:


To watch videos at 60 fps, switch to YouTube at 1080p60.

If you do it right, and your computer and web browser are capable of showing video at 60 fps, then the video below should be played smoothly and without any brakes. If this is not the case, then this is why we are talking about this — many other applications also demonstrate this behavior, not just games. For now, I can only recommend that you try to watch the video on some other machine, or just read the text.


Check, check, one, two, three ... You should see this video in smooth 60 fps.

And now let's get down to business. If you encounter brakes, then most likely it looks like this:


This is what the brakes at 60 fps look like. We call this symptom a cardiogram.

Yes, this is exactly what the brakes look like, even when the game is running at 60 fps. You could encounter something similar in any modern game, and probably thought that “the game is not optimized.” So, you should reconsider your theory (that such brakes arise from the "slow" rendering of the game). If the game is “too slow,” it means that at some moments it will not be able to render one frame fast enough, and the monitor will have to re-show the previous frame. Therefore, when we record a video of such a game at 60 fps, we see “missing frames”. (These are frames for which the next frame was not rendered on time, so the current frame is shown twice.)

Now rerun the previous video with brakes (“cardiogram”), pause it and press the .(dot) key in the YouTube player to move from frame to frame. Try to notice when the same frame is shown twice. Come on, try, I'll wait ...

---

Well, found? Not? Strange, isn't it? ..

The video does not look smooth when you watch the whole animation as a whole, but when you look frame by frame, there are no gaps !

How can this be?

Let me explain in more detail. Here is a comparison of perfectly smooth video and video with brakes in the form of a "cardiogram" , reproduced at 1/20 of the original speed, so that we can see the individual frames:


Top correct video with 60 fps, bottom "cardiogram". Playback is slowed down 20 times.

Two things can be noticed: first, they actually reproduce with the same frequency - when there is a new frame from above, there is a new frame from below. Secondly, for some reason, they move a little differently - there is a noticeable “gap” in the middle of the image, which then becomes more or less noticeable.

An attentive viewer may notice one more curious detail: the bottom one - the braking one - the image - which is considered “slow” ... in fact, it “outstrips” the correct one. Strange, right?

If we look at the two adjacent frames and their timings (note that in all the videos I showed there were exact timers (with an accuracy of 1/10 000 seconds), then we can notice something very interesting: the first two frames are perfectly synchronized, but the third ...


Six consecutive frames from the video with precise timings. The upper ones are correct, the lower ones are with a “cardiogram” .

... in the third frame, we see that the tree in the “braking” video is significantly ahead of its copy from the correct video (circled in red). You may also notice that this frame seems to take longer (circled in yellow).

Wait ... if the video is “slower”, and the shot “took longer”, then how can it stay ahead of the right one?

To understand this, you need to understand how in our time games and other three-dimensional interactive applications implement animations and rendering. (Experienced developers should forgive me for bothering them with obvious things, but I need to be sure that everyone reading this text can understand the text.)

Brief history of frame timing

Once upon a time, in a distant galaxy far away ... When developers created the first video games, they usually adjusted to the exact frame rate at which the display works. In the NTSC regions, where televisions operated at 60 Hz, this meant 60 fps, in the PAL / SECAM regions, where televisions operated at 50 Hz, this meant 50 fps. They could not even think about any possibility of “skipping a frame”.

Most of the games were very polished and simplified concepts that worked on specific equipment - usually on an arcade computer, or on a " home microcomputer ", like ZX Spectrum , C64 , Atari ST , Amstrad CPC 464 , Amigaetc. In fact, the developer created the design, implemented and tested the game for a specific machine and a specific frame rate, and was 100% sure that she would never miss a single frame.

The speeds of the objects were also stored in units of “frames”. Therefore, it was said, not how many pixels per second the character should move, but how many pixels per frame . It is known that in Sonic The Hedgehog for Sega Genesis, the rotational speed, for example, was 16 pixels per frame . Many games even had separate versions for the PAL and NTSC regions, in which the animations were manually drawn specifically for 50 fps and 60 fps. In fact, the launch at any other frame rate was not even considered.

When the games started working on more diverse machines — in particular, on PCs with expandable and replaceable hardware, the developers could no longer know at what frame rate the game would now work. Complement this with the fact that the games have become more complex and unpredictable. The most outstanding in this regard were the 3D-games, differing in the variety of scenes complexity, sometimes even depending on the player. For example, everybody likes to shoot at barrels with fuel - a huge explosion occurs, beautiful effects ... and the inevitable drawdown of personnel. But here, no one is against passing frames, because it is so interesting.

Therefore, it was very difficult to predict how much simulation and rendering of one frame would take. (It’s worth noting that on modern consoles, the hardware is still unchanged, but the games themselves are still often unpredictable and complex.)

If we cannot be sure at what frame rate the game will work, we have to measure the current frame rate and constantly Adapt the physics and speed of game animations. If one frame takes 1/60 second (16.67 ms), and the character runs at a speed of 10 m / s, then in each frame it moves 1/6 meter. But if the frame ceases to take 1/60 of a second, but instead suddenly began to take 1/30 of a second (33.33 ms), then you need to start moving the character 1/3 meter (twice “faster”) per frame so that screen, he continued to move with apparent constant speed.

How does the game do it? In fact, it measures the time at the beginning of one frame, and then at the beginning of the next, and calculates the difference. This is a fairly simple way, but it works very well. So sorry, he worked well. In the 90s (remember the phrase “35 fps was considered a decent value for serious network battles” from the beginning of the article), people were more than satisfied with this method. But at that time, the graphics card (do not forget, they were not even called the GPU ) was a very "thin" element of hardware, and the main CPU directly controlled the objects on the screen. If the computer did not have a 3D accelerator, the processor itself even painted these objects. So he knew exactly when they would appear on the screen.

What happens today

Over time, we had more complex GPUs, and they became more and more " asynchronous ". This means that when the CPU gives the GPU a command to draw something on the screen, the GPU simply saves this command to the buffer so that the CPU goes about its business while the GPU performs rendering. As a result, this led to the fact that when the CPU tells the GPU that “this is the end of the frame,” the GPU simply saves the message as another piece of data. But he does not treat him as something particularly urgent. Yes, and how he can - after all, he still needs to process some of the previously transferred commands. He will show the frame on the screen when he finishes all the work he was given earlier.

Therefore, when the game tries to calculate the timings by subtracting the time stamps of the beginning of two frames, then the relevance of this, roughly speaking ... is very doubtful. Let's go back to our example from short videos. We have these shots where the camera moves along the trees:


Six consecutive frames from the video with precise timings. The upper ones are correct, the lower ones are with a “cardiogram” .

Now remember what I said about timings and movements. In the first two frames, the frame timing is 16.67 ms (that is, 1/60 second), and the camera moves by the same amount from the top and bottom, so the trees are synchronized. In the third frame (below), the game saw that the frame time is 24.8 ms (which is more than 1/60 second), so she thinks that the frame rate has decreased and is in a hurry to move the camera a little further ... just to find out that the timing is The fourth frame is only 10.7 ms, so the camera moves here a little slower, and the trees are more or less synchronized again. (Synchronization will be fully restored only after two frames.)

Here the following happens: the game measures what it considers to be the beginning of each frame, and these frame timings sometimes fluctuate for various reasons, especially on such a high-loaded multitasking system as the PC. Therefore, at some moments the game thinks that it does not have time to provide 60 fps, and at some moments it generates animation frames calculated for a lower frame rate. But due to the asynchronous nature of the work of the GPU, he actually manages to provide 60 fps for each individual frame of this sequence .

This is what we perceive as brakes - the animation generated under the varying frame rate (“cardiogram”) is actually displayed with the correct constant frame rate.

That is, in fact, there is no problem here - everything really works smoothly, just the game does not know about it.

In this case, we return to the beginning of the article. Finally, having discovered the cause of the problem (in fact, it is an illusion of a problem - because it really is not there, is it?), This is what we did to check:


First we see the cardiogram, and then use a little trick to get rid of it.

In the first part of the video, from the very beginning you can see the problem with the "cardiogram" . Then we change the “magic” option, after which everything becomes perfectly smooth!

What is this magic option? In Serious Engine, we call it sim_fSyncRate=60. In simple terms, it means “to completely ignore all these problems with timings and pretend that we always measure stable 60 fps”. And thanks to this, everything works smoothly - simply because everything worked smoothly anyway! The only reason why everything seemed to be retarding was the erroneous timings used for animation.

And that is all? Is it enough to do this and everything will be great?

Is the solution so simple?

Unfortunately not. This is just a developer test. If we stop to measure the frame rate in real situations and will simply assume that it is always equal to 60, then when it falls below 60 - and the PC sooner or later, be sure to fall for one reason or another: the OS will start some background process energy saving or GPU / CPU overheating protection will turn on ... who knows, everything will slow down.

So, if we change the frame rate, then braking occurs, if not, then everything can slow down at some moments. What do we do?

The real solution is to measure not the beginning / end of rendering a frame, but the time when the image is displayed on the screen.

How can we tell the game that the image of the frame is already displayed on the screen? You may be surprised to learn this - but today there is no such way!

This is shocking, I understand. It could be expected that this is the basic function of each graphic API. But it turns out that in the process of gradual improvements here and there everything, in essence, missed the very meaning of the problem. We all forgot about the subtle details of our work, continuing to repeat the same thing, and the graphic APIs have evolved in all other aspects, except for this: the application cannot in any way know for sure that the frame is actually displayed on the screen. We can find out when its rendering will end, but not when it will be shown.

What's next?

But do not worry, everything is not so gloomy. Many people in the graphic ecosystem are actively working to implement support for proper frame timing for different APIs. For the Vulkan API , there is already an extension called VK_GOOGLE_display_timing, which has demonstrated its usefulness in implementing the proof of concept. However, it is available only for a limited range of equipment, mainly on Android and Linux.

Work is already underway to create similar and better systems for all major graphics APIs. When will it end? It is difficult to say, because the problem lies quite deep inside various OS subsystems.

However, I assure you that Croteam works tirelessly to ensure that this problem is resolved as soon as possible, and that all participants in the interactive graphics ecosystem understand and support our efforts.

We aim to make this solution available to a wider audience, and when this happens, we will prepare The Talos Principle update with the implementation of this function.

Various clutter and other details

The last phrase can be considered the end of the main text. Below are the "bonus sections", almost not related to each other and the main text. Probably, I will update them when the situation changes, or in the case of more complex issues that need to be resolved in the near future.

"Composer"

Frosted glass effect? Yes, for him we definitely need a composer. He just needs it, right?

Behind the scenes, a concept called “composite window manager” , or composer , is connected with all of this . It is a system that now exists in all operating systems, which allows windows to be transparent, have a blurred background, shadows that pop up over the Skype window, etc. Composers can even display windows in 3D. To do this, the composer takes control over the last stage of creating a picture of a frame and decides what to do with it before it reaches the monitor screen. This complicates things even more.

In some operating systems, the composer can be disabled in full screen mode. But this is not always possible, and even if it is possible - then how can you be banned from running the game in windowed mode?

Power and heat management versus rendering complexity

We must also take into account that modern CPUs and GPUs do not operate at a fixed frequency, but have systems that change their speeds in accordance with the load and temperature. Therefore, the game cannot simply assume that the GPU and the CPU will have the same speed in each frame. On the other hand, the OS and drivers cannot expect that the game will have the same amount of work in each frame. To take this into account, it is necessary to develop complex systems for communication between these two parties.

Can't we just ...

Most likely no. :) Usually, GPU time measurement is considered an alternative to display timings. But it does not take into account the presence of the composer and the fact that none of the GPU rendering timers is actually synchronized directly with the display update. For perfect animation, we absolutely need to know the time to display the image, not the time to complete the rendering.