Self-deception of simplicity

The article considers the problem of exceeding the terms of creating programs as a result of self-deception: as if programming is simple, and even simpler. The creators of new technologies in the field of software development assure that their technology will help to solve a huge number of problems and that now software development is becoming easier and faster than ever. But practice is inexorable. Again and again, the deadlines for completing the next program project are disrupted.

Introduction

Failure of software development timelines is a common occurrence in the practice of creating software products. This phenomenon is harmful to all participants in the process, from ordinary developers who are forced to work in a state of emergency, and ending with management and customers who are forced to put up with lags and additional costs.

There are many reasons for this, but in this article I want to express the opinion that the main component is the widespread belief that software development is easy. Immediately emphasize an interesting point. In general, everyone understands that creating programs is a laborious process that requires qualification and time, but as soon as it comes to discussions or presentations, everyone, pursuing their own interests, begins to say that all the difficulties are in the past and now, using the new technology / programming language / methodology, everything will become easy and simple. This has been repeated over and over for many years. In general, there are no improvements, and the deadlines are still breaking down. It is the causes of these phenomena and their consequences that we will discuss in this article.

Why did I think about these issues

I will proceed not from general reasoning, but from personal experience. I am creating a tool for programmers PVS-Studio. This is a static C / C ++ code analyzer for checking programs written for 64-bit systems or created using OpenMP technology. In such programs, a number of error patterns arise, which, however, are easy to diagnose by analyzing the source code. But that's not the point. The fact is that, while participating in the promotion of these tools, I came across a lot of resistance caused by the fact that the use of 64-bit technologies and OpenMP is presented as an extremely simple thing. As a result of this confidence, there is a need to overcome the enormous resistance of people who believe that there are no problems and cannot be. I seemed to be the enemy of their calm world. A similar picture, of course,

The belief that everything is simple is not surprising. All around describe how easily you can make your program 64-bit and immediately get a performance boost and a huge amount of available memory. It describes how, using OpenMP, you can easily turn your code into parallel. Around articles containing theses in the spirit of "enough to recompile the application," "a simple set of OpenMP directives." And on this wonderful holiday of life I go with a poster, with large headlines for the articles “20 traps of porting C ++ code to a 64-bit platform”, “32 OpenMP pitfalls when programming in C ++”. At the same time, I feel like a harmful nasty old man who is dissatisfied with everything and constantly grumbles at everything. It was this strange sensation and resistance to the “evangelists of simplicity” that made me think and write this text.

There was a moment when I pondered the thought that perhaps I was actually wrong. We need to talk with everyone how simple it is. And that with our tools, everything is simple at all. But something is wrong. There is a contradiction. I have a good idea of ​​the situation, and I am confident that there are difficulties waiting for programmers to master parallel technologies and 64-bit systems with a large amount of memory. There is no such simplicity. This is a lie. And deception is harmful. It is harmful to me, as a person interested in promoting their tools. It is harmful to the leader, who cannot correctly predict the timing. And, of course, he is harmful to the programmer, who is waiting for disappointment in new technologies and overtime. Therefore, I decided to stay in my positions and try to change your point of view as well.

Where does the difficulty come from

I can still try to convince me that creating 64-bit programs is not so difficult, that a number of programs can be rebuilt for 64-bit systems without corrections. And errors can be automatically found and corrected using different techniques. Although this will not be easy, as I have experience talking about the complexity of these processes and how easy it is to make mistakes in estimating deadlines several times.

But when parallel programming is also presented, this is already a complete discrepancy between reality. Parallel programming is a complex and too complicated task that cannot be solved by simply placing OpenMP directives around loops or using some kind of library. But that’s how it is presented. A lot of developers who have not yet studied the creation of parallel algorithms are told that they don’t need to think about it and that OpenMP or some other technology will correct the situation. From such articles, it seems simple and understandable how to parallelize any program. But the popularization of what the reader cannot do and check beforehand creates only the illusion of understanding. Perhaps she enriches his vocabulary and somehow expands his thinking,

This does not mean that OpenMP or any other technology is bad. They are wonderful. But you need to understand what they can and what not. Do not expect to easily parallelize your existing programs. Most likely, this will require the creation of new algorithms and the transformation of data structures and mechanisms for working with them. The ease of incorporating OpenMP into existing code should be understood so that this integration will entail fewer changes than using other technologies. So MPI technology makes it easier to build a distributed system, compared to implementing the same functionality through direct work with APIs and network protocols. In turn, MPI is unnecessarily complex for multi-core systems with shared memory and requires significant changes in the code. Compared to it, OpenMP is really extremely easy to integrate.

It is impossible to talk about the simplicity of parallel programming, no matter what technologies are used. This area is complex in itself. If you want to make sure, then try to quickly sit down yourself and parallelize the array sorting algorithm that everyone wrote at school and institute. If that doesn't work, then look at Butcher's parallel sorting. Can’t you imagine? You can’t imagine. This example is made to show what kind of difficulties you should expect even with the usual algorithms used in your programs. But there are many other aspects.

You probably use iterators in loops in your C ++ programs? Do you know that in popular articles about OpenMP, when they show examples with parallelization for, you forget to warn that iterators cannot be used in it? Indexes should be simple data types (e.g. int). That is, when planning time for alteration of algorithms, keep in mind that you may also need to change data structures and methods of working with them. And all this is extra time and effort. Yes, OpenMP 3.0 has the ability to work with iterators. Articles prior to OpenMP 3.0 circumvented the topic of their use. New articles are written in the style of "it became easier to write programs on OpenMP 3.0, due to the possibility of using iterators." Yes, it’s good that now there is such an opportunity. But now imagine the number of nuances and difficulties that exist,

If everything is complicated, then why say it is simple?

There are no villains who want to trick you and convince you that everything is simple. But there are various interests that lead to mutual deception. Anyone who promotes 64-bit hardware platforms and operating systems is naturally interested in convincing that it is extremely simple to transfer your applications to them. This is clear. The transition to 64-bit systems has already dragged on, and if we say that there are difficulties, this can extend the transition by a few more years.

Manufacturers of parallel systems advertise various software technologies and libraries, convincing that everything will become parallel almost by itself. Those who create these libraries / technologies try to convince programmers that their library / technology is the best, as it is the simplest. And here there is a sad confusion. The fact that the library is simple and convenient (for example, in my opinion, OpenMP is simple and convenient) does not mean at all that creating parallel programs is also easy!

Programmers, having read a lot of articles about how everything is fine in the 64-bit and parallel worlds, but not yet having practice, deceive the management and themselves about the deadlines. Management cheats customers. And the management itself likes to consider everything simple, as it seems to them that this allows saving on developers. As a result, there are no perpetrators, but all are deceiving each other.

conclusions

Already in the process of writing this text, I found Dijkstra's remarkable article “Two Views on Programming” [1], which in many ways intersects in meaning with this article. I will quote a small excerpt that summarizes all of the above very well: “Basically, this is not the fault of computer manufacturers who want to do business as if they are selling simple products; and it’s not the fault of software project managers who prefer to view programmers' activities as a simple and predictable process; and it’s not the fault of educational institutions that would like to prepare students for guaranteed success. This is a consequence of the comfortable illusion that Man is only a complex automaton, an illusion that, like a drug, brings its victims an apparent release from the burden of responsibility.

But the weight of the burden should not be the reason why you should continue to keep your eyes closed. Everyone loses from this. One must accept the complexity as it is.

The conclusion from all this should be a more critical look at the capabilities of the latest technologies. New tools and techniques are useful, but they could never and may not be able to replace the very process of developing software, which consists of many stages and elements. First of all, it is necessary to learn and improve the development process, and only then choose new convenient technologies for the project.

Indispensable assistants in getting out of the circular "deception of simplicity" are the wonderful books by Steve McConnell "How much does a software project" [2] and "Perfect code" [3]. And, of course, the iconic book by Frederick Brooks, “The Mythical Man of the Month, or How Software Sets Are Created” [4].

Bibliographic list

1. Edsger Dijkstra. Two views on programming. URL http://www.inr.ac.ru/~info21/pdf/dijkstra.pdf
2. McConnell S. How much does a software project cost. - M.: “Russian Edition”, St. Petersburg: Peter, 2007. - 297 pp., Ill.
3. McConnell S. The Perfect Code. Master class / Per. from English.- M.: Publishing House "Russian Edition", St. Petersburg: Peter, 2007.- 896 pp., ill.
4. Frederick Brooks. Mythical man-month or how software systems are created. - Per. from English - SPb .: Symbol-Plus, 1999, - 304 p .: ill.