Is it hard to develop software?
One of these days, I started talking with a friend about the fact that there is a development process, are programmers working or are they pushing bugs? Can anyone become a developer? What is the complexity of the software development process? I hope our dialogue will help to understand the matter for those who are not involved in programming or just like to reflect on their favorite pastime.
And it all started with a fairly popular question.
- Why is the project development budget often exceeded?
- This is a difficult question. I propose to take as a basis one metaphor that will help me explain the situation. Of course, this is my opinion, based on the experience that I received.
So, the developers are the creators of new worlds. Virtual worlds.
- You would have told me about the film The Matrix!
- Actually a metaphor from there. So here. The input requirements give an idea of what kind of world the customer needs (the customer can also be inside the company).
Most often, the requirements are broad and blurry.
- Wait a minute. Why is it immediately blurry? For example, I have a project that I want to run. I clearly understand how it should work. Moreover, I have TK !!!
- Hah. Yes, but the customer does not think and should not think about many aspects of this new world, because this is the work of the developers. How to organize the source code? What variable naming convention to follow? Which libraries to use? What branching strategy should I use? How to ensure performance?
By and large, the task of the developer is to lay such laws in this new world so that this world solves the original problem.Relate business or functional requirements to the requirements of the laws of the new world. Moreover, this world must constantly change under the influence of new requirements and information.
- Suppose, but what does this have to do with the budget and timelines?
- Let's think about what affects the amount of time, and therefore the budget, necessary to create a new feature on the project?
Well, there is a certain feature. There is time to figure out how to do it. To implement and test it. And let's say that the developer has all the information on how to do it - he knows perfectly the necessary technologies, programming languages and approaches. He understands the requirements - he knows how it should work in the end.
Will the task implementation time be equal to the time of writing the code? Of course not. In addition to all this, the developer must embed a new feature in an existing world. Embed in existing laws. Add new laws or change existing ones so that both old and new features work.
This is precisely the most difficult, because the complexity of these issues is most often many times higher than the complexity of implementing the feature itself.
- This is how to build a house? For example, if I want to add another window to the room, I have to consider if there is enough heating? Will I break the laws of heat transfer?
- Exactly. Now imagine that heat transfer is just one aspect. In complex software systems, there are hundreds, if not thousands, of such aspects. Imagine, when implementing each small new feature, you need to pay attention to all aspects. And only now I can try to explain the main idea. As the creation and development of such worlds (projects), the number of laws and rules inside is constantly growing. It is not so simple to track that some law of this world is outdated and needs to be disposed of.
The world is overgrown with unnecessary rules. Members of the development team begin to live in a more complex world, their labor costs are increasing.
- And why it is impossible to gradually revise what was done and simplified, remove unnecessary?
- So it is possible. There are even specific practices - continuous refactoring and so on. But self-reflection at the team level is rarely possible, especially at the technical level. Yes, and finding all the problems is difficult - the size of the projects is more and more. You can only watch so that the architecture during the implementation of specific tasks does not get worse.
As a result, the developer lives with the understanding that the task is difficult to solve - after all, many connections need to be rebuilt in this fantasy world of his. Grades are overpriced. Tasks really take longer than they could.
It’s like running with weights - much more energy is spent on covering the same distance.
- Another scenario - the developers do not overstate, because they do not see the increasing complexity, although it is. There are already laws in their world, they simply do not know them. And as in real life - ignorance of the laws does not exempt from responsibility. This leads either to the accumulation of technical debt - there are problems, but they will come up later. Or the implementation of new features leads to the collapse of other parts of the system, and this may not be immediately noticeable - you do not always double-check all the work and something will pop up later (which essentially increases the cost of a specific feature). Oh yes, you say autotests - but is it cheap to develop them? Do they test 100% performance?
- Come on, it seems to me you're pumping - but what about modularity? Even I am not a programmer, I understand how to deal with complexity. Divide into parts. Separate areas of responsibility.
- You're right. That is what most modern programming practices are aimed at. For example, maybe you heard about microservice architecture. Developers are trying to reduce the complexity of systems by limiting the amount of responsibility.
But the problems remain the same, just part of them moves to another plane - integration. Establishing links between the same modules or services.
I wanted to give you another example about the previous thesis - the difficulty of amending existing laws.
I like to philosophize on this topic. Why is it difficult to see the need for change?
When you created the world, and not just know its laws, they were born by you, it is difficult to perceive things differently. To expose your logic to strong criticism yourself - this is how one can reach schizophrenia. And redrawing these laws can be very difficult. Not always long, but difficult, because it requires a change in some paradigm of this world in the head of a particular developer or the whole team. Imagine that you need to rethink everything in the real world if the understanding of the law of gravity changes. Or, for example, suddenly, did you find out that the earth is still flat? These are such basic concepts and they are so woven into our lives that the process of rethinking can take a lifetime. And essentially nothing really will change except your perception of this world. Therefore, developers do not like to change their paradigms. In any case, the majority.
- It seems I'm starting to understand. Of course, it’s difficult to immediately try on yourself and realize all this complexity just by talking with you, but at least I began to understand more the specifics of the work. Perhaps today I have enough food for thought, but I will gladly continue our dialogue at the next meeting ...
And it all started with a fairly popular question.
- Why is the project development budget often exceeded?
- This is a difficult question. I propose to take as a basis one metaphor that will help me explain the situation. Of course, this is my opinion, based on the experience that I received.
So, the developers are the creators of new worlds. Virtual worlds.
- You would have told me about the film The Matrix!
- Actually a metaphor from there. So here. The input requirements give an idea of what kind of world the customer needs (the customer can also be inside the company).
Most often, the requirements are broad and blurry.
- Wait a minute. Why is it immediately blurry? For example, I have a project that I want to run. I clearly understand how it should work. Moreover, I have TK !!!
- Hah. Yes, but the customer does not think and should not think about many aspects of this new world, because this is the work of the developers. How to organize the source code? What variable naming convention to follow? Which libraries to use? What branching strategy should I use? How to ensure performance?
By and large, the task of the developer is to lay such laws in this new world so that this world solves the original problem.Relate business or functional requirements to the requirements of the laws of the new world. Moreover, this world must constantly change under the influence of new requirements and information.
- Suppose, but what does this have to do with the budget and timelines?
- Let's think about what affects the amount of time, and therefore the budget, necessary to create a new feature on the project?
Well, there is a certain feature. There is time to figure out how to do it. To implement and test it. And let's say that the developer has all the information on how to do it - he knows perfectly the necessary technologies, programming languages and approaches. He understands the requirements - he knows how it should work in the end.
Will the task implementation time be equal to the time of writing the code? Of course not. In addition to all this, the developer must embed a new feature in an existing world. Embed in existing laws. Add new laws or change existing ones so that both old and new features work.
This is precisely the most difficult, because the complexity of these issues is most often many times higher than the complexity of implementing the feature itself.
- This is how to build a house? For example, if I want to add another window to the room, I have to consider if there is enough heating? Will I break the laws of heat transfer?
- Exactly. Now imagine that heat transfer is just one aspect. In complex software systems, there are hundreds, if not thousands, of such aspects. Imagine, when implementing each small new feature, you need to pay attention to all aspects. And only now I can try to explain the main idea. As the creation and development of such worlds (projects), the number of laws and rules inside is constantly growing. It is not so simple to track that some law of this world is outdated and needs to be disposed of.
The world is overgrown with unnecessary rules. Members of the development team begin to live in a more complex world, their labor costs are increasing.
- And why it is impossible to gradually revise what was done and simplified, remove unnecessary?
- So it is possible. There are even specific practices - continuous refactoring and so on. But self-reflection at the team level is rarely possible, especially at the technical level. Yes, and finding all the problems is difficult - the size of the projects is more and more. You can only watch so that the architecture during the implementation of specific tasks does not get worse.
As a result, the developer lives with the understanding that the task is difficult to solve - after all, many connections need to be rebuilt in this fantasy world of his. Grades are overpriced. Tasks really take longer than they could.
It’s like running with weights - much more energy is spent on covering the same distance.
- Another scenario - the developers do not overstate, because they do not see the increasing complexity, although it is. There are already laws in their world, they simply do not know them. And as in real life - ignorance of the laws does not exempt from responsibility. This leads either to the accumulation of technical debt - there are problems, but they will come up later. Or the implementation of new features leads to the collapse of other parts of the system, and this may not be immediately noticeable - you do not always double-check all the work and something will pop up later (which essentially increases the cost of a specific feature). Oh yes, you say autotests - but is it cheap to develop them? Do they test 100% performance?
- Come on, it seems to me you're pumping - but what about modularity? Even I am not a programmer, I understand how to deal with complexity. Divide into parts. Separate areas of responsibility.
- You're right. That is what most modern programming practices are aimed at. For example, maybe you heard about microservice architecture. Developers are trying to reduce the complexity of systems by limiting the amount of responsibility.
But the problems remain the same, just part of them moves to another plane - integration. Establishing links between the same modules or services.
I wanted to give you another example about the previous thesis - the difficulty of amending existing laws.
I like to philosophize on this topic. Why is it difficult to see the need for change?
When you created the world, and not just know its laws, they were born by you, it is difficult to perceive things differently. To expose your logic to strong criticism yourself - this is how one can reach schizophrenia. And redrawing these laws can be very difficult. Not always long, but difficult, because it requires a change in some paradigm of this world in the head of a particular developer or the whole team. Imagine that you need to rethink everything in the real world if the understanding of the law of gravity changes. Or, for example, suddenly, did you find out that the earth is still flat? These are such basic concepts and they are so woven into our lives that the process of rethinking can take a lifetime. And essentially nothing really will change except your perception of this world. Therefore, developers do not like to change their paradigms. In any case, the majority.
- It seems I'm starting to understand. Of course, it’s difficult to immediately try on yourself and realize all this complexity just by talking with you, but at least I began to understand more the specifics of the work. Perhaps today I have enough food for thought, but I will gladly continue our dialogue at the next meeting ...