Imaginary problems cause bad software
The fact that they are interesting to solve does not mean that they are needed by someone.
“A group of people brainstorming over a laptop and a piece of paper”, photo by Stefan Stefanchik with Unspalsh
There are many factors that lead to the creation of bad software: the choice of tools, communication in a team, developers' personal disinterest in success, testing methodology. It seems to me that all this has a main root cause: these are imaginary problems.
Overly complex or non-functioning software was not designed as such. It is simply designed for something else, and not for a real task.
Suppose you publish podcasts, and you need a website where you can sell related products, make money from advertisements and, most importantly, publish podcasts, videos and blogs.
Requirements for this small web application might look something like this:
- Fast download for North America with live podcast broadcasts
- No failures in the first 15 minutes for 99.99% of users, preferably a complete absence of failures
- Good integration with Google Adwords and possibly other third-party advertising systems if you have time.
- Dynamic links to the latest products in the shop and, if possible, recommendations to users based on the pages viewed
- Integration with the Facebook Live player. If you can easily make an alternative solution for streaming without Facebook, then even better
You give these specifications to the contractor team and talk a little with them. It seems all on the same wavelength. But when they show a prototype two months later, your face is poured with blood. You spent $ 15 thousand on a piece of garbage - and you want to get your money back.
When you first open the application freezes. You ask how to choose the type of banners - and you are shown an ugly, incomprehensible UI. Half of the links to the goods in the shop are broken or there are no images, and the live broadcast via Facebook lags!
But the development team does not understand your anger. In fact, from their point of view, they managed to do a hell of a difficult job.
They put their hearts into creating the application, it has amazing possibilities:
- The most advanced recommendation system
- Algorithm of text decoding of all audio streams in real time
- Home page loads faster than 200 ms around the world
- The streaming protocol and client are made almost from scratch if you don’t want to rely on Facebook Live
- Developed a service that allows you to easily integrate more than 20 advertising exchanges
The problem is that you requested a base product with several additional features, provided that they are fairly simple to implement. And the development team understood otherwise. They heard some exciting tasks to tackle ... and many boring, basic functions that are not worth much attention and rigorous testing.
Worse, you didn’t communicate directly with the developers - you communicated through a damaged phone. You spoke with the sales person who held meetings with some middle managers. They wrote some business specifications for a project manager who wrote technical specifications — and gave them to the team leader or architect, and he and his team began developing the product. Each link introduced distortions along the way.
Imaginary problems are often more interesting to solve than real ones. People with an abundance of intelligence play competitive games, invent and solve math problems, and write books to answer abstract questions about human nature — all for free, just for fun. But a mediocre programmer will probably charge you a hefty sum for creating a simple Android application. This is not because mediocre programmers are harder to find than geniuses, but because the first activities are interesting, and the latter are pretty boring.
Most programmers want to get money and have fun at the same time. Of course, the definition of “interesting” is different for everyone, but for many engineers it comes down to solving interesting and complex problems that are in the field of solvability.
Give an intelligent person too many boring tasks that cannot be automated, and you will eventually drive him crazy. However, the human brain after billions of years of evolution has become very resourceful in maintaining the mind. Just as victims of child deprivation or ill-treatment find salvation in science-fiction books, victims of corporate programming and freelancers seek salvation in solving imaginary problems.
The number of imaginary problems that a software engineer can create for himself depends on his imagination and the complexity of the real problems that need to be solved.
It should be noted that this problem is not unique to developers. Management, sales, HR, support, legal and even accounting departments - all have their own unique ways of creating imaginary problems. Some try to get into the decision-making process when their presence at the meeting is merely a formality or not at all required. Others overestimate the petty problem associated with their role, or hire a lot more employees than necessary to demonstrate their importance.
When problems are too stupid, smart people will find a way to fix the situation.
But imaginary problems come not only from bored developers. They are also generated as a result of long chains of communication.
When I first started looking for clients as a freelancer, I could not afford to build communication at my own discretion. So I had to deal with long mail threads with hundreds of letters, where minor details of internal MVPs were discussed. People within a week changed each requirement in the project. I had clients who asked such questions: “Will it be compatible with an ICO?” Or “Can I add some AI here?”
Of course, most clients have enough experience, but even they often lack the knowledge to formulate or build some requirements. This is normal, since part of my work as a “computer scientist” is to help people understand what they are doing and what they do not need, based on their particular situation. But it is much more difficult to determine when there are several gaskets between you and the customer.
Requirements change, because someone either misunderstood the intention, or tried to cope with the aforementioned boredom.
Most companies like to start a sales person in the staff who sticks to potential customers, bargains and generally describes the product. There are also specialists with advanced communication skills to discuss with the client more detailed specifications and requirements: this is usually the same sales person, only with a slightly different job title. And there is an internal corporate system, different levels of management, and perhaps some hierarchy in a technical team.
When a list of customer requirements passes through so many people, even if these people have the best intentions, something is inevitably lost in the process. Sometimes this is because the original requirement did not make sense or needs to be changed. Perhaps the sales manager said to the client: "In total for 39 999 above, we will do it on the blockchain." He agreed, and all the rest remains to guess what it means to “do on the blockchain.”
More often than not, the requirements change either because someone either misunderstood the intention, or tried to cope with the aforementioned boredom, trying to make his work or the work of his team more interesting and impressive.
Behind all this, the initial requirements are lost - those are real problems that need to be solved. They are replaced by imaginary problems and voids, and you have a lot of people ready and willing to fill these voids with their imaginary problems, because the real problems that they have to solve are boring, and filling the voids gives them satisfaction.
Excessive complexity and natural selection
Often there is a darker reason for the appearance of imaginary problems: such problems help a team or company grow, they can even become an integral part of its work.
“People who are taken out are taken away and encouraged to look for complex solutions, there is no incentive to introduce simplified ones”
- Nassim Nicholas Taleb
Have you heard about those three programmers who have found a really simple solution for secure web banking? They developed impeccable banking software from scratch using functional design methodology and secure languages, and then began to transfer large banks to their amazing infrastructure.
Perhaps you have not heard of them, because they do not exist. But there are many teams of thousands of developers who are not able to assimilate simple concepts, such as "rolling back to the old version" , and they are constantly busy creating banking software.
Storing and transferring numbers is not a particularly difficult problem. Here, indexing the entire contents of the Internet and issuing the corresponding result for a query in natural language in a split second is another matter.But only a few smart guys managed to solve this problem.
A chronic problem with online banking is that the banking ecosystem has indeed reached perfection in its own money appropriation hierarchy. Its leaders are corrupt leeches that cling to the body of society, but the leaders of the organization only reflect the state of the entire system.
I do not say that most ordinary bank employees are evil and harmful personalities. Far from it. As a rule, these are friendly guys who earn money for food, accommodation and education for their families. But their main incentive is not to fix banking software, but to save work. Loss of work in the modern economy is a serious problem for some people, and in the banking industry a careless word or excessive initiative is an easy way to appear before a disciplinary committee.
Thus, banking systems remain the same - not because they are effective, but because of inertia. This inertia manifests itself in the form of solving imaginary problems in order to avoid solving real problems - the solution of which, as already noted, threatens the work of other people. Solving these real problems can lead to dismissal or, in the case of some particularly unpleasant "institutions" like Goldman Sachs, to several letters that ruin the life of a former employee and end in strange suicide .
“It’s hard to make a person understand something if his salary depends on what he doesn’t understand!”
- Upton Sinclair
Ordinary houses ignore the fact that top management spends 90% of their time on “meeting customers”, including traveling to tropical islands and millions of budgets for “overhead”. In turn, top management closes eyes to corruption in the rank and file.
As middle managers encourage their wolf-wall-street fantasies, top management turns a blind eye to the behavior of these managers, who buy eccentric offices, hire three secretaries and a dozen interns.
Since ordinary managers do not complain about their dictatorial fantasies and lust for power, middle-level managers turn a blind eye to the fact that instead of cutting costs, they spend time presenting PowerPoint on “Improving our agile Agile methodology.”
Since the Timlides don't pay attention that their bosses don't even know how to use Excel correctly and come into the office a couple of times a week, ordinary managers turn a blind eye when Timlides and architects talk about “the next generation of interaction between our systems through JRPC and microservice using Hibernate and Spring, when these damn MySQL queries are executed for more than a day.
This is a vicious circle of solving imaginary problems: from a CEO who does not understand that stealing another 30 million will not give him parental love, to an intern who does not understand that the new “Send” button on Angular-Material-Bootstrap 19.13.5 will not change that passwords are stored in plain text (and used to check cookies).
But everyone has to continue to solve imaginary problems, because if they stop doing this, they will begin to focus on real problems - and they will understand that the whole system is broken. They can understand that in the corner of the office, Debra has been looking at the uptime schedules of the server cluster for ten years, although the company moved to AWS five years ago. They can understand that 99% of their tasks are needed only to save someone else's work. And this awareness is difficult to accept, I dare say, even impossible for most. Therefore, most find a way to adapt.