Imaginary problems are the root of bad software.
- Transfer
There are many circumstances that can be catalysts for creating bad software: the tools used, the quality of communication in a team, the personal qualities of developers, methodologies, etc. And there is one thing among them that is the root of almost everyone else: imaginary problems.
Most of the sophisticated or buggy software was not planned to be complicated and, even more so, zagugovanym. It was simply created to perform not the tasks that were the basis of the original idea.
Let's imagine that you are recording a popular podcast and at some point thinking about creating your own website - well, you know, with information about your podcast, records of past issues, articles, and perhaps some advertising. In fact, how much profit can be shared with some outside publishers?
And so you decide to hire people who will make this site for you. You write absolutely clear requirements for them:
You give these requirements to the developers and, perhaps, communicate with them a little. It takes 2 months. They show you demos and you are covered in red spots. It becomes clear that you just threw $ 15,000 into the abyss. What you were shown is completely unacceptable from any side, just a bunch of garbage. You want your money back, but the train has already left.
Getting angry at the sight of a demonstrated site is very simple. At the first opening, everything just stopped. Then it seemed to work and you asked how to change the type of advertising that will be displayed on the site. You were shown the out-of-the-bicycle-bike interface for this, which you can’t learn on your own. Broadcasts from Facebook inhibit. Everything is terrible.
But the development team does not understand why you are upset. From their point of view, they accomplished the feat, having made such a complex product in just 2 months. They have invested in it all their talent, their whole soul. Judge for yourself what great features they added to it:
You already see what the trouble is, right? You asked for a very specific, highly specialized product with a couple of additional features you need. The development team heard it differently. For them, all your “desirable”, “more convenient” and “in the future” have turned into an exciting task of developing a common, expandable, scalable and complementary product of enormous proportions, during the implementation of which you can heroically solve a bunch of interesting tasks. And, of course, there is nothing to be distracted by such trifles as polishing and bringing to the ideal the basic functionality of the product - we have what scale, it is not the time now to trade it out!
Another problem is that you most likely did not communicate directly with the direct developers of the system. You played a spoiled phone: talked to some sales manager who transferred tasks to someone from the middle management of your company, wrote business specifications there, gave them to the project manager, wrote technical specifications and gave them to the team leader, who broke them into tasks and distributed to developers. Well, each of the developers also understood and realized their tasks in the context of their understanding.
Imaginary problems are more interesting to solve than real ones. Very smart people play complex games, solve math problems, write books on abstract topics - for free or almost for free. At the same time, the average programmer for a completely standard mobile application will charge you quite a substantial bill. This is not because the average programmer is harder to find than genius. This is because interesting tasks are easy and pleasant to do, but routine is not.
Most programmers want to work on interesting tasks and get good money for it. To achieve this is difficult. Of course, you can speculate on what an “interesting task” is, but for most developers it is something that is very close to the boundary of the field of theoretically solvable problems. Something difficult to achieve, but possible.
Give a reasonable person a lot of routine, non-automated tasks, and sooner or later you will bring it to the handle. The human brain, however, after millions of years of evolution, has developed quite a few different tricks to maintain itself in an adequate state. As victims of violence are able to escape into the world of their fantasies, so innocent to the years of enterprise-development or web freelancing eventually find salvation in solving imaginary problems.
The number and complexity of invented problems are functions of the programmer’s imagination level and the complexity of his real-life tasks. It should be noted that this approach itself is not unique to software developers. Managers, salespeople, HR, support, lawyers and accountants find their unique ways of creating and heroically overcoming non-existent ills. They involve themselves in decisions that are beyond their competence, speak at meetings, where their presence is a mere formality, and so on. They also overestimate the scale of the problems and their role in solving them (“Our new Dog Diary app should be 101% compatible with GDPR from the very first day! We cannot wait for the customer to complain!”). They inflate the staff to create the appearance of the importance of the work of their team, and then deal with the problems of growth,
Many of the developers are smart, but many of the tasks assigned to them are pretty dumb. This contradiction makes smart people invent themselves other, albeit not existing in reality, but interesting tasks.
Sometimes imaginary problems are not the result of fantasies of bored developers. Sometimes this is the result of a “damaged phone”.
I sometimes work on freelancing. When I first started, I could not afford to sort out customers. This meant the need to take everyone in a row and observe in all colors the most diverse manifestations of deviations of the human psyche. I saw chains of hundreds of letters on the subject of completely irrelevant details of the prototype. I have seen people who change every single paragraph of specification every week. I had clients who, for projects of some trivial sites, asked about the possibility to immediately go with them to ICO or to tie artificial intelligence to them.
There were also quite sensible customers who, however, lacked the knowledge (or vocabulary?) To express all their requirements in clear language. This is not a problem in itself. Part of the work of software developers once again is the collection of requirements and writing specifications. This is something for which we, including paying money, can even do this work more or less normally, if you have normal access to the client and enough time. The trouble is that sometimes there is no such access and there are several intermediaries between you.
Most companies like to have salespeople in selected dedicated positions. These are special people who are looking for potential customers, discussing tasks, deadlines and prices with them. It is these people who can find out the most useful and ask the most questions. But these are not the people who will write this project. Between salespeople and programmers there are 2, 3, 4 or more layers of middle and lower managers, architects, analysts and team leaders. Yes, it can be very skilled people. But still - these are additional layers. No matter how hard they try, information passing through them will change. Someone will drop something (in his opinion) unimportant, another will clarify something unspecified, but (in his opinion) obvious, the third will replace the foolish formulation with the correct (as he is sure) term - and now you will not recognize the original task. The salesman can throw a phrase like “and for 39 999 above we can do it on the blockchain”. If the client bites, the development team in two levels below will have to wrestle with it, and how is it possible to do this on the blockchain. But after all it is already sold and paid, we will do.
Thus, the original problem is lost in speculation and rethinking. There are holes between new, invented problems (they wouldn’t even gap there) and there will be people who want to fill them with their own imaginary problems and their brilliant solutions. Because it is still freedom, not a routine.
Often there is an even darker side. Coming up with imaginary problems and methods to solve them can become a growth driver for a company, and in the future, perhaps its main business, the most important part of its existence, which cannot be abandoned.
“People selected to solve complex problems have no incentive to solve simple ones” - Taleb
Have you heard about those three engineers who suddenly found out that online banking is a fairly simple thing? They created simple and therefore reliable software, used the correct methods and programming languages, and then transferred all the main banks to their platform.
No, have you heard? This is because this was not. But here's what was, is and will be - separate teams of thousands of programmers working for different banks and for some literally half a year are able to implement such a feature as a “rollback of a transaction” by the forces of the whole team. That's how banking software is written, yes.
And this, of course, is not because it is so difficult to move a number from one column to another, no. It is not difficult. That is to index the entire Internet, and after that give relevant search results for a split second - this is difficult. However, a couple of guys in the garage did it. And for a rollback transaction you need 1000 people and six months.
The problem here is that the banking ecosystem is doing an incredibly good job with its main task - supporting the banking ecosystem. The wheels are spinning, the paper is rustling and tomorrow everything will be the same as it was yesterday. Everything has inertia. And, if we have been working on an imaginary problem for the last 5 years, we will also work on it tomorrow.
“It’s hard to make a person understand something if he gets his salary for not understanding it” - Upton Sinclair
And everyone continues to work on imaginary tasks, because if they stop for a moment and look at the real ones, it will become clear that their whole system is broken. Suddenly, it may turn out that Vasya, sitting in that corner, has been monitoring the state of the internal server farm for the last 10 years, from which AWS successfully migrated 5 years ago. Suddenly it turns out that all of Masha's job is to send Dasha's reports to someone. Such awareness is very difficult and people unconsciously try to avoid situations in which they can be made by chance.
And we continue to solve imaginary problems.
Most of the sophisticated or buggy software was not planned to be complicated and, even more so, zagugovanym. It was simply created to perform not the tasks that were the basis of the original idea.
The story of podcasts
Let's imagine that you are recording a popular podcast and at some point thinking about creating your own website - well, you know, with information about your podcast, records of past issues, articles, and perhaps some advertising. In fact, how much profit can be shared with some outside publishers?
And so you decide to hire people who will make this site for you. You write absolutely clear requirements for them:
- Fast loading site in North America
- Support downloads of past podcast releases and live streaming of current ones.
- The broadcast should not fall or freeze during the first 15 minutes for 99.99% of users. It is desirable to never, but at least so.
- Integration with Google Adwords (and possibly with peers in the future)
- Integration with Facebook broadcasts, since there you make your broadcasts. If you can create an alternative solution that will allow you to stream more conveniently - even better.
You give these requirements to the developers and, perhaps, communicate with them a little. It takes 2 months. They show you demos and you are covered in red spots. It becomes clear that you just threw $ 15,000 into the abyss. What you were shown is completely unacceptable from any side, just a bunch of garbage. You want your money back, but the train has already left.
Getting angry at the sight of a demonstrated site is very simple. At the first opening, everything just stopped. Then it seemed to work and you asked how to change the type of advertising that will be displayed on the site. You were shown the out-of-the-bicycle-bike interface for this, which you can’t learn on your own. Broadcasts from Facebook inhibit. Everything is terrible.
But the development team does not understand why you are upset. From their point of view, they accomplished the feat, having made such a complex product in just 2 months. They have invested in it all their talent, their whole soul. Judge for yourself what great features they added to it:
- A delightful recommendation system
- Speech Recognition Algorithm that displays subtitles under your broadcast in real time !!!
- Home page of the site is loaded for 200 ms anywhere in the world
- Broadcast protocol and client are written from scratch, and Facebook is added to them by the plugin. At any time, you can add plug-ins for other platforms!
- There is a possibility of integration with 20 different advertising platforms.
You already see what the trouble is, right? You asked for a very specific, highly specialized product with a couple of additional features you need. The development team heard it differently. For them, all your “desirable”, “more convenient” and “in the future” have turned into an exciting task of developing a common, expandable, scalable and complementary product of enormous proportions, during the implementation of which you can heroically solve a bunch of interesting tasks. And, of course, there is nothing to be distracted by such trifles as polishing and bringing to the ideal the basic functionality of the product - we have what scale, it is not the time now to trade it out!
Another problem is that you most likely did not communicate directly with the direct developers of the system. You played a spoiled phone: talked to some sales manager who transferred tasks to someone from the middle management of your company, wrote business specifications there, gave them to the project manager, wrote technical specifications and gave them to the team leader, who broke them into tasks and distributed to developers. Well, each of the developers also understood and realized their tasks in the context of their understanding.
Reality avoidance mechanism
Imaginary problems are more interesting to solve than real ones. Very smart people play complex games, solve math problems, write books on abstract topics - for free or almost for free. At the same time, the average programmer for a completely standard mobile application will charge you quite a substantial bill. This is not because the average programmer is harder to find than genius. This is because interesting tasks are easy and pleasant to do, but routine is not.
Most programmers want to work on interesting tasks and get good money for it. To achieve this is difficult. Of course, you can speculate on what an “interesting task” is, but for most developers it is something that is very close to the boundary of the field of theoretically solvable problems. Something difficult to achieve, but possible.
Give a reasonable person a lot of routine, non-automated tasks, and sooner or later you will bring it to the handle. The human brain, however, after millions of years of evolution, has developed quite a few different tricks to maintain itself in an adequate state. As victims of violence are able to escape into the world of their fantasies, so innocent to the years of enterprise-development or web freelancing eventually find salvation in solving imaginary problems.
The number and complexity of invented problems are functions of the programmer’s imagination level and the complexity of his real-life tasks. It should be noted that this approach itself is not unique to software developers. Managers, salespeople, HR, support, lawyers and accountants find their unique ways of creating and heroically overcoming non-existent ills. They involve themselves in decisions that are beyond their competence, speak at meetings, where their presence is a mere formality, and so on. They also overestimate the scale of the problems and their role in solving them (“Our new Dog Diary app should be 101% compatible with GDPR from the very first day! We cannot wait for the customer to complain!”). They inflate the staff to create the appearance of the importance of the work of their team, and then deal with the problems of growth,
Many of the developers are smart, but many of the tasks assigned to them are pretty dumb. This contradiction makes smart people invent themselves other, albeit not existing in reality, but interesting tasks.
The architecture of the "spoiled phone"
Sometimes imaginary problems are not the result of fantasies of bored developers. Sometimes this is the result of a “damaged phone”.
I sometimes work on freelancing. When I first started, I could not afford to sort out customers. This meant the need to take everyone in a row and observe in all colors the most diverse manifestations of deviations of the human psyche. I saw chains of hundreds of letters on the subject of completely irrelevant details of the prototype. I have seen people who change every single paragraph of specification every week. I had clients who, for projects of some trivial sites, asked about the possibility to immediately go with them to ICO or to tie artificial intelligence to them.
There were also quite sensible customers who, however, lacked the knowledge (or vocabulary?) To express all their requirements in clear language. This is not a problem in itself. Part of the work of software developers once again is the collection of requirements and writing specifications. This is something for which we, including paying money, can even do this work more or less normally, if you have normal access to the client and enough time. The trouble is that sometimes there is no such access and there are several intermediaries between you.
Most companies like to have salespeople in selected dedicated positions. These are special people who are looking for potential customers, discussing tasks, deadlines and prices with them. It is these people who can find out the most useful and ask the most questions. But these are not the people who will write this project. Between salespeople and programmers there are 2, 3, 4 or more layers of middle and lower managers, architects, analysts and team leaders. Yes, it can be very skilled people. But still - these are additional layers. No matter how hard they try, information passing through them will change. Someone will drop something (in his opinion) unimportant, another will clarify something unspecified, but (in his opinion) obvious, the third will replace the foolish formulation with the correct (as he is sure) term - and now you will not recognize the original task. The salesman can throw a phrase like “and for 39 999 above we can do it on the blockchain”. If the client bites, the development team in two levels below will have to wrestle with it, and how is it possible to do this on the blockchain. But after all it is already sold and paid, we will do.
Thus, the original problem is lost in speculation and rethinking. There are holes between new, invented problems (they wouldn’t even gap there) and there will be people who want to fill them with their own imaginary problems and their brilliant solutions. Because it is still freedom, not a routine.
Artificial complexity and natural selection
Often there is an even darker side. Coming up with imaginary problems and methods to solve them can become a growth driver for a company, and in the future, perhaps its main business, the most important part of its existence, which cannot be abandoned.
“People selected to solve complex problems have no incentive to solve simple ones” - Taleb
Have you heard about those three engineers who suddenly found out that online banking is a fairly simple thing? They created simple and therefore reliable software, used the correct methods and programming languages, and then transferred all the main banks to their platform.
No, have you heard? This is because this was not. But here's what was, is and will be - separate teams of thousands of programmers working for different banks and for some literally half a year are able to implement such a feature as a “rollback of a transaction” by the forces of the whole team. That's how banking software is written, yes.
And this, of course, is not because it is so difficult to move a number from one column to another, no. It is not difficult. That is to index the entire Internet, and after that give relevant search results for a split second - this is difficult. However, a couple of guys in the garage did it. And for a rollback transaction you need 1000 people and six months.
The problem here is that the banking ecosystem is doing an incredibly good job with its main task - supporting the banking ecosystem. The wheels are spinning, the paper is rustling and tomorrow everything will be the same as it was yesterday. Everything has inertia. And, if we have been working on an imaginary problem for the last 5 years, we will also work on it tomorrow.
“It’s hard to make a person understand something if he gets his salary for not understanding it” - Upton Sinclair
And everyone continues to work on imaginary tasks, because if they stop for a moment and look at the real ones, it will become clear that their whole system is broken. Suddenly, it may turn out that Vasya, sitting in that corner, has been monitoring the state of the internal server farm for the last 10 years, from which AWS successfully migrated 5 years ago. Suddenly it turns out that all of Masha's job is to send Dasha's reports to someone. Such awareness is very difficult and people unconsciously try to avoid situations in which they can be made by chance.
And we continue to solve imaginary problems.