Thoughts on modern C ++ and game development

Published on January 05, 2019

Thoughts on modern C ++ and game development

Original author: Ben Deane
  • Transfer
The new year for game developers began with a wave of criticism that struck the C ++ Standardization Committee after the publication of Aras Prankevichus “Complaints about modern C ++” . There was a serious question: did the standards committee really lose touch with reality, or was it the other way around, and did this game developers break away from the rest of the C ++ community?

We offer you a translation of Ben Dean ’s popular post , a veteran of the gaming industry who has worked for a long time at Blizzard, Electronic Arts and Bullfrog as a C ++ developer and team leader, in which he responds to criticism from the perspective of his own experience.

TL; DR: The C ++ Standardization Committee does not have a hidden goal to ignore the needs of game developers, and “modern” C ++ is not going to become an “non-debugable” language.
Throughout last week , Twitter was actively debated , during which many programmers - especially those who work in the field of game development - said that the current vector of development of "modern C ++" does not meet their needs . In particular, from the standpoint of the usual game developer, it looks as if the debugging performance in the language is ignored, and the code optimization becomes expected and necessary.

Due to the fact that in 2019 I had worked in the gaming industry for more than 23 years, I have my own opinion based on observations on this topic in relation to game development, which I would like to share. Is "debugging" important for game developers and why? What are the issues related to it?

For a start - a small excursion into history.

Many game developers who write in C ++ work in Microsoft Visual C ++. Historically, a huge gaming market has emerged around the Microsoft platforms, and this is reflected in the typical experience of the average game programmer. In the 90s and 2000s, most games were written in the light of these circumstances. Even with the advent of consoles from other manufacturers and the growing popularity of mobile gaming, the assets of many AAA studios and numerous game programmers today are Microsoft-made tools.

Visual Studio is probably the best C ++ debugger ever. And most of all, Visual Studio really stands out precisely in terms of debugging programs — more than with its front-end, back-end, STL implementation, or anything else. In the past five years, Microsoft has made significant progress in developing tools for developing C ++, but even before these achievements, the debugger in Visual Studio has always been very cool. So when you are developing on a Windows PC, you have a world-class debugger at hand.

Considering the above, let's consider the process of obtaining a code in which there will be no bugs; the possibilities we have from the point of view of a programmer who does not play games; and the limitations that game developers face. If we rephrase the main argument in favor of the “vector of development of modern C ++”, then it will be reduced to types, tools and tests. Following this thought, the debugger should be the last line of defense . Before we reach it, we have the following options.

Option # 1: Types


We can use as much typing as we need to eliminate whole classes of bugs during compilation. Strong typing, no doubt, is an opportunity that the recent evolution of C ++ has given us; for example, starting with C ++ 11, we managed to get:

  • significant expansion type traits;
  • such innovations, as nullptrwell as scoped enumto combat the legacy of C - weak typing;
  • GSL and auxiliary tools;
  • concepts in C ++ 20.

Some of you may not like template metaprogramming; others may not like the way they write code, which is almost universally used auto. Regardless of these preferences, the main motive for using the listed styles in C ++ is clearly visible here - this is the desire to help the compiler so that he, in turn, can help us, using what he knows best: a type system.

If we talk about game programming, strong typing here is a wide field for research, and it is actively used by familiar game programmers who are interested in improving their C ++ skills in practice. There are two important things of concern here: the effect on compile time, and the effect on code readability.

Frankly, you can easily ignore the compilation time - but only if you are a programmer in a very large company that does not play games and has a well-established internal infrastructure and endless computing power to compile any code you can write. . Such large companies are concerned about the cost of compilation - therefore they use modules - but, as a rule, this does not cause pain to individual developers. At the same time, for most game programmers, this is not at all the case. Indie developers don't have farms to build builds; AAA game developers often use something like Incredibuild, but, given the fact that they can easily work with a code base that has turned 10 years old or more, the assembly process can still take 15-20 minutes.

We can argue about the relative cost of adding “hardware” versus the time cost of a programmer, and I agree with the position that hardware costs less, however:

  • The hardware is the real one-time expenses that will be borne by the budget of the current quarter, as opposed to not so tangible expenses in time / hiring / and the like, which will be distributed over a longer period of time. People do not cope well with the decision in favor of such a compromise, and companies are specially built in such a way as to optimize short-term profit.
  • Infrastructure requires support, and almost no one goes into the gaming industry in order to become a release engineer. Compared to other areas where C ++ is used, the salary of game developers is not so high - and non-game engineers are paid even less.

You can also speculate on the fact that the compile time should never have reached such a state; and again I agree with you. The price of this is in constant vigilance - outgoing, again, from the release engineer - and, ideally, some automated tool that allows you to track changes over time required to build the build. Fortunately, due to the emergence of CI-systems, this goal today can be achieved much easier.

Option # 2: Tools


We must use the maximum of tools available to us — warnings, static analysis, sanitizers, dynamic analysis tools, profilers, and others.

My experience says that game developers use these tools where possible, but here the industry as a whole has several problems:

  • These tools tend to work better on non-Microsoft platforms - and, as mentioned earlier, this is not a typical scenario in game development.
  • Most of these tools are designed to work with “standard” C ++. They are supported out of the box std::vector, but not my own class CStaticVectorfrom a hypothetical engine. Of course, blaming the tools for this is useless, but it is still one of the barriers to their use that developers have to overcome.
  • Creating and maintaining a CI chain that will launch all of these tools requires the presence of release engineers — and, as mentioned earlier, hiring people for engineering jobs that are not directly related to games is a systemic problem for the gaming industry.

So, since these tools work so well with standard C ++, then why don't game developers use STL?

How to start the answer to this question? Perhaps, from the next excursion into the history of game development:

  • Until the early 90s, we did not trust the C compilers, so we wrote games in assembly language.
  • From the beginning to the mid-90s, we began to trust the C compilers, but we still did not trust the C ++ compilers. Our code was C, which used C ++ style comments, and we no longer needed to write typedefs for our structures all the time.
  • Around 2000, the C ++ revolution occurred in the game development world. It was the era of design patterns and large class hierarchies . At that time, STL support on consoles left much to be desired, and the world was then ruled by consoles. On PS2, we are forever stuck with GCC 2.95.
  • Around 2010, two more revolutions are being undertaken. The pain of using large class hierarchies has stimulated the development of a component code approach. This change continues its evolution today in the form of Entity-Component-System architectures. Hand in hand with this was the second revolution - an attempt to take advantage of multiprocessor architectures.

In the course of these paradigm shifts, the gaming development platforms themselves were constantly changing, and moreover they changed seriously. Segmented memory has given way to a flat address space. Platforms have become multiprocessor, symmetric and not very. Game developers, accustomed to working with Intel architectures, had to get used to MIPS (Playstation), then to a special “hardware” with heterogeneous CPU (PS2), then to PowerPC (XBox 360), then to even greater heterogeneity (PS3) ... Each New platform came new performance for processors, memory and disks. If you wanted to achieve optimal performance, then you had to rewrite your old code, a lot and often. I’m not even going to mention how much the appearance and growth of the Internet, as well as the limitations

Historically, STL implementations on gaming platforms have been unsatisfactory. It is not a secret that STL-containers are poorly suited for games. If you press the game developer to the wall, then perhaps he admits that std::string- quite a OK, and std::vector- a reasonable option by default. But all the containers contained in the STL have a problem of controlling allocation and initialization. In many games, you have to worry about the limitation of memory for various tasks - and for those objects, the memory for which you will most likely have to allocate dynamically during gameplay, slab or arena allocators are often used . Amortized constant time- not a good result, since the allocation is potentially one of the most “expensive” things that can happen during the execution of the program, and I don’t want to miss a frame only because it happened when I did not expect it. I, as a game developer, have to manage my memory requirements in advance.

A similar story is obtained for other dependencies in general. Game developers want to know what takes each processor cycle, where and when and for what each byte of memory is responsible, as well as where and when each execution thread is monitored. Until recently, Microsoft's compilers changed the ABI with every update - so if you had a lot of dependencies, rebuilding all of them could be a painful process. Game developers usually prefer small dependencies that are easily integrated, do just one thing and do it well — preferably with a C-style API — and are used in many companies, are in the public domain, or have a free license that is not requires indication of the author. SQLite and zlib- good examples of what game developers prefer to use.

In addition, the C ++ games industry has a rich history of patients with the “Not invented here” syndrome. This is to be expected from an industry that began with enthusiastic singles who were making something of their own on completely new equipment and had no other options. The gaming industry, among other things, is the only one where programmers appear in captions in no particular order. Writing a variety of things is fun, and it helps your career! It is much better to build something of your own than to buy ready-made!And since we are so worried about performance, we can tailor our solution so that it is appropriate for our project - instead of taking a generalized solution that wastes resources available. Dislike Boost is a prime example of how such thinking manifests itself in game development. I worked on projects that went the following way:

  • To begin with, to solve this or that problem, we connect the library from Boost to the project.
  • Everything works very well. When you need to upgrade, there is a little pain, but no more than when updating any other dependencies.
  • Another game wants to use our code, but the stumbling block is that we use Boost - despite the fact that our experience with using Boost was quite normal.
  • We remove the code using Boost, but now we are faced with a new problem: we have to solve a problem that was previously solved instead of our library from Boost.
  • We essentially copy the parts of the Boost code we need into our own namespaces.
  • Later, we inevitably and again and again encounter the fact that we need additional functionality, which would already be in the original code, if we had not thrown it out. But now we ourselves are the owners of this code, so we have to continue supporting it.

We don’t like something huge trying to do too many things at the same time or that can affect compile time — and that’s quite reasonable. What people are mistaken over and over again is that they are opposed to accepting the supposed pain today - while because of this decision they are faced with a very real and much greater pain supported by something the budget they will have to experience over the next three years. Alas, but the presence of evidence in the form of games that successfully use a dish from STL and Boost, in no way can affect the psychology of a person and persuade game developers.

For all these reasons, many gaming companies have created their own libraries that cover what STL does — and more — while supporting game-specific use cases. Some large gaming companies were even able to master the development of their own, fully-fledged, almost completely compatible with the STL replacement API , which later resulted in huge costs for supporting this project.

It is reasonable to find an improved alternative std::map , or use small buffer optimization in std::vector. It is much less acceptable to be doomed to support your own implementation algorithmsortype traitsthat will bring almost no benefit. As for me, it is regrettable that STL for most developers is only containers. Since learning STL at the start is taught by him, then speaking about STL most implies std::vector- although in fact they should be thinking about std::find_if.

Option number 3: Tests


It is argued that extensive testing should be carried out, TDD and / or BDD should cover all the code that can be covered, and bugs should be fought by writing new tests.

So let's discuss the topic of testing.

Judging by my experience, automated testing in the gaming industry is almost never used. Why?

1. Because correctness is not so important, and there is no real specification.


As a young programmer in the gaming industry, I quickly got rid of the thought that I should strive to model something realistically. Games are the smoke and mirrors and the search for short confuses. No one cares how realistic your simulation is; The main thing is that it is fascinating. When you have no other specification than “the game should be felt right,” there is no test item itself. Thanks to the bugs, the gameplay can even get better. Quite often, the bug gets into the release, and even wins the love of users ( remember the same Gandy from Civilization ). Games are different from other areas in which C ++ is used; here the lack of correctness does not lead to the fact that someone eventually loses their savings.

2. Because it's hard


Of course, you would like to do automated tests wherever you can. This can be implemented for some subsystems for which there are clearly defined outcomes. Unit testing in the gaming industry, of course, is present, but is usually limited to low-level code — the previously mentioned STL analogs, string conversion procedures, physics engine methods, etc. Those cases where the executable part of the code has predictable results are usually tested by unit tests, although TDD is not used here - because game programmers prefer to simplify their lives, and not vice versa. But how do you test the gameplay code (see point one)? As soon as you go beyond unit testing, you are immediately confronted with another reason why testing games is so difficult.

3. Because content is involved in it.


Testing of non-trivial systems may include the provision of content, with the participation of which it will be carried out. Most engineers are not very good at making this content on their own, so you need to involve someone with the right content creation skills to get a meaningful test. After that, you will encounter the problem of measuring what you get at the output - it’s no longer a string or a number, but an image on the screen or a sound that changes over time.

4. Because we do not practice it.


Unit testing is a function for which I know the possible inputs and outputs. However, gameplay is an unpredictable, dynamically developing behavior, and I don’t know how such a phenomenon could be properly tested. What I can test is - if, of course, I get permission from my manager to devote enough time to it - this is, for example, performance, or high-level matchmaking capabilities, which I can analyze. Such infrastructure work can be fascinating for some game programmers, but for the majority it is simply uninteresting - and, in addition, requires the approval and support of the wallet owner. In the role of a game programmer, I never have the opportunity to practice writing high-level tests.

5. Поскольку [компания] не видит необходимости в автоматизированном тестировании


Our main goal is to release the game. We live in the times of an industry that moves forward with hits that earn most of their money in the first month of sales, when the cost of marketing these hits is at its maximum. The life cycle of consoles taught us that the code will not live in such a long time either. If we are working on an online game, then most likely we will get additional time to test matchmaking or server load. Because for the release of the game we need its performance to be in order, we should at least do performance testing, but we should not automate this process. For management in the games industry, automated testing is nothing more than a waste of time and money. For it, you have to hire experienced engineers who will do the work, the result of which will be almost imperceptible. The same time could be spent on developing new features. In the short term, it is much more profitable to use QA personnel to test the game, which brings us to the next point.

6. Because in general, testing refers to second-rate activities in games.


I adore good QA specialists. For me they are worth their weight in gold. They know how to make your game better, breaking it in a way that would never occur to you. They are profile experts in your gameplay in the sense that you do not understand, and hardly ever will understand. They are better than a team of super-capable compilers that help you do everything right. I am glad that I had the chance to work with several great QA specialists over the years of my work.

I almost always had to fight only to keep them on my team.

In large AAA companies, a QA organization is usually a completely separate department from any development team, with its own management and organizational structure. This is supposedly done so that they can show objectivity during testing. In practice, everything is far from perfect.

These are treated like gears in a huge mechanism, which are often thrown between projects without warning and generally treat them as if anybody can handle their work. When the project “moves out” from the deadline, engineers can feel the crunch the hard way, but QA gets much stronger, because they have to work on the night shift and on weekends, plus they also get it for bringing sad news about the current quality of the project.

They are seriously underpaid. The most experienced testers with years of expertise in the subject area receive less than half of what they pay to the average developer. I had to work with the smartest QA engineers who created pipelines for performance testing with tracking and alerts, created frameworks for testing APIs and for load testing, and performed many other tasks supposedly unworthy of the time of "real engineers." I am sure that these smartest people would get much more if they worked in any other big technology company.

They do not trust. It is not uncommon that testers are kept apart from other developers, and their badges allow them to gain access only to that floor of the building where they work themselves - or even use a separate entrance.

They are forced to obey. Testers are often told not to disturb other engineers. When they need to report a bug directly, they are asked to contact the engineers with respect, like "Mrs. X." or "Mr. Y.". Sometimes I got a call from irritated QA-department heads - in those cases when I contacted those who discovered the bug directly for a joint investigation.

All this sounds like a terrible fairy tale, and let not everyone has to deal with such things, unfortunately it still happens quite often; so often that engineers begin to think — perhaps themselves under the burden of constant stress, but this does not excuse them — that the job of QA is to look for their own bugs, or, even worse, they begin to blame QA for the bugs.

In the best teams with whom I had to work, we insisted that our teams had their own QA engineers who would work with us together. However, they did not lose their objectivity or desire to achieve a better result. They were pleased to receive help from programmers in writing automated tests. What I do not doubt for sure is that it would be useful for the gaming industry to do automation more often.

Debug performance


With all this in mind, the habits of debugging, the platform for APIs and tools that are still growing up, and the complexity (combined with a lack of culture) of automated testing, it becomes clear why game developers insist on debugging.

But there are still problems with debugging itself, and problems with how game developers cope with the current vector of C ++ development.

The main problem with debugging is that it does not scale. Among the game developers of developers reading this post, there are those who decide that the phenomena I described do not agree with what they observed in their practice. Quite possibly, this is due to the fact that sooner or later they themselves had to face the problem of debugging scalability, and they found a way around it.

In other words, we want to have a productive debugging, because in order to catch bugs, we often need to be able to run applications with fairly large and representative data sets. But in fact, when we reach this point, the debugger usually becomes too rough a tool to use, regardless of whether it is productive or not. Of course, setting breakpoints on data (data breakpoints ) can be useful for catching medium-sized problems, but what to do if we run into real bugs — those that remain after we seemingly fixed everything? With those that arise under load in the network, or in the event of a lack of memory, or working at the limit of multi-threading capabilities, or occur only for a small, not yet identified subset against a million other players, or occur only on disk versions of the game, or only in the assembly in German, or after three hours spent on stability testing ( soak testing )?

Like hell, we can rely on only one debugger. In this case, we do what we have always done. We try to isolate a problem, make it happen more often; we add logging and sift our program through it; we adjust the timers and thread settings; we use binary search by builds; we study core dumps and crash logs; we are trying to reproduce the problem, cutting content to a minimum; we reflect on what might be causing the problem, and discuss it.

Often, until we get to the real cause of "crash", we will have time to fix a few other things. In other words, we solve problems, and in the end, using a debugger is only a small part of this process. So yes, debugging speed is a nice addition, but its lack does not prevent us from continuing to be engineers. We still need our other skills, such as the ability to analyze core dumps and read an optimized assembler.

When using "modern C ++" I use the debugger in the same way as usual. I go through them in the freshly written code; I put breakpoints on the data that interests me; I use a debugger to investigate unfamiliar code. With the advent of “modern C ++”, none of this changes, and yes, even though STL uses _ Ugly _Identifiers, it does not make STL magic. Sometimes it is useful to see what STL does “under the hood”, or step over it; or, as you can now do, use the debugger to hide the library code from me .

When I encounter debugging performance problems, it's usually not that “modern C ++” slows me down - the fact is that by now I’m already trying to do too much. Using the debugger does not scale - unlike types, tools and tests.

I myself was concerned about the problem that C ++ code requires more and more optimization, and I was interested in the opinions of compiler developers on this issue. The fact is that there is no definite answer. We are already in the continuum, and we have the opportunity to move further in this direction without harming the ability to debug the code. Today, our compilers perform copy elision (skip copy) for temporary objects., even if we do not ask them to perform this optimization. This does not affect our ability to debug applications. I doubt that we will complain that the debug builds include NRVO or another half a dozen optimizations that can be done in such a way that we will not notice them during the debugging. I suspect that C ++ is moving in that direction.

Epilogue: The Path of Modern C ++


If you work as a programmer in the field of game development and you do not like where C ++ moves, then you essentially have two options for possible further actions.

1. Do nothing


Assuming that you are still going to write code in C ++, then you can just continue to use the language in the same way you did before. There is no need to start using any new features if you do not want to do this. Virtually all of what you are using now will continue to be maintained — and in the years to come you will continue to reap the rewards of improving the compiler.

This is a completely adequate behavior strategy for those who work for themselves or with a team of like-minded people. C ++ 98, along with some newer features, is still well suited for writing games on it.

However, if you work in a large company, then sooner or later you will have to face changes in the language, since you will have to increase the team and hire new people. In turn, when you hire C ++ developers, this will mean hiring developers on “modern” C ++. There will be a change of generations - as it already happened with the assembler, C and C ++ 98. You can control the process if you put restrictions on what is allowed in your code base and what is not, but this decision will not save you in the long term. And what do you do in this case?

2. Take part


Instead of going only one GDC once a year, start visiting CppCon , where you will benefit much more from the money your company spent on a ticket. Participate in discussions of standards; join groups and subscribe to newsletters; read draft standards and provide authors with feedback. If you can also attend committee meetings, it will be just fine, but even if not, you can still do a lot to get your point across to others.

Participation in the C ++ committee is open to all. All the necessary information for those who want to take part in the work of SG14, or SG7, or SG15 - or any other working group relating to your area of ​​interest -can be found at isocpp.org . The committee has no secret plans - in fact, do you really think that over 200 programmers can agree on a single agenda? Here, even the “bosses” of the committee often fail to “shove” their ideas.

If you want your opinion to be heard, then you should start talking where your opinion can be heard, not on Twitter or Reddit. Please take advantage of this advice - I look forward to our discussion.