10 bugs leading to software over-engineering
- Transfer
- Tutorial
Several things are guaranteed to increase over time: distances between stars, the entropy of the universe, and business software requirements. Many articles write "Do not complicate!", But do not write why or how to do it. Here are 10 clear examples.
We engineers consider ourselves the smartest people. Well, since we create different things. And this error often leads to over-engineering. If you have planned and built 100 modules - Business will always ask you for the 101st, which you never thought of. If you gather your strength and solve 1000 problems, they will come to you and lay out 10,000 new ones on the table. You think that everything is under control, but in fact you can’t even imagine which direction the road will lead you tomorrow.
In my 15 years as a programmer, I have never once seen a Business give complete and stable software requirements once and for all. They always change, expand. And this is the nature of the business, not the mistakes of the people who manage it.
Moral : Casino (business) always wins.
When the Business throws us more and more demands (as expected), we sometimes say to ourselves: “Ok, let's try to group and summarize everything that is possible!”.
This is why most MVC systems end with a Big Model or a Fat Controller. But, as we have already found out, business requirements will never stop being added, which means that this is the way to nowhere. In fact, we need to respond to this like this:
Example: once for one of the customers we developed a user profile module. We started with the classic CRUD controller, because this is just a user profile and everything should be simple there. It all ended with the implementation of 13 different scenarios for creating and filling out this profile: registration through a social network, regular, simplified, quick with subsequent editing, looking completely different for subsites, etc. They had very little in common with each other. The scenarios “viewing an order” and “editing an order” are similarly different.
For starters, try dividing business functionality vertically before dividing it horizontally. It even makes the task of dividing the monolith into microservices easier. Sometimes it even becomes monolithic at all, or at your services. Otherwise, it becomes more and more difficult to change parts of your system.
Moral : Prefer isolated actions, avoid combining.
Tip : Take some user-visible part of your program (form \ page \ command). How many context switches does a programmer need to understand all the code associated with it?
(Something in common with the previous paragraph, but sometimes it happens independently of it)
Sometimes engineers bring. Instead of solving business problems, they spend time choosing the right parent class. And the answer is simple.
Software architecture always plays catch-up with business requirements. So even if with the help of some magic you find the perfect abstraction today, its expiration date will immediately come with it - see point No. 1 above - business always wins. So the best measure of the build quality of your architecture is how quickly it can be taken apart. There is a great article on how to write code that will be easy to remove, not easy to change.
Moral : Code duplication is better than incorrect abstraction.
Moreover, code duplication sometimes helps to find that very correct abstraction. Because when you see several parts of a system using the same code in the same way, you can understand what unites them. The quality of Abstraction is determined by the quality of its weakest link. Duplication of the code allows you to look at different situations from different angles and see the boundaries of Abstraction more clearly.
There is a practice of writing wrappers (wrappers) for each external library used (mainly because of following the style of the main product, sometimes because of the reasons described in the previous two paragraphs). A very popular approach in enterprise programming. Programmers in dynamic languages like Node / Ruby / Python simply add a library and start using it. But the Enterprise Developer cannot go for it - he needs to create a wrapper. And then another wrapper over the wrapper. Perhaps this made some sense in the 2000s, when most of the libraries were a mishmash, and in general there was not much open source code.
But today is the 2016th year. External libraries improved by an order of magnitude. They became just fantastic. Most likely they were written by excellent programmers and tested better than your main code. They have a distinct API. You can embed logging into them. You do not need to spend your time writing wrappers around already good code. In addition, most wrappers are completely meaningless. Their interface is very closely related to the library below, often simply in the form of a reflection of one-to-one functions. If the library changes in the future, most of the code using this wrapper will also have to be changed. Creating an “agnostic” wrapper that can remain unchanged even, for example, with the complete replacement of the library to be wrapped with another, is not a trivial task. Such tasks are sometimes posed with the thought of the potential "configurability" of the general solution,
Morality : wrappers should be exceptions, not the norm.
Mindlessly following code quality concepts (like all variables must be declared as “private final”, each class must have an interface) will not make your code AUTOMATICALLY good. Check out, if you haven’t seen it yet, the enterprise version of FizzBuzz or Hello World . Breakthrough code. At the micro level, each class follows SOLID principles and uses great patterns. If you set some code quality analysis tool on this code, it will say that it is a wonderful product, a sight for sore eyes. But if you take a step back, you will immediately see what a nightmare this program is, just printing Fizz Buzz.
Moral : Always take a step back and look at the big picture.
Likewise, automated tools define the test coverage of the code well, but they say absolutely nothing about whether you are testing what you need or not. They can measure performance in some test, but they won’t say how well your program handles data in another case. They all look at the micro level: what does this class \ method \ string do. And only Man looks at the big picture.
Learn a different programming language and try different approaches to solve problems you already know. This will make you a significantly better programmer. Do not get hung up on just one thing.
Let's take some closely related functionality and divide it into 10-20 layers, where each layer has no meaning without the rest. Well, because we want to implement the concept of “Testable code”, or “The principle of shared responsibility”, or call it something else fashionable. In the past, this was done through inheritance. Class A inherits from B, which inherits from C, etc.
Today, people are doing the same thing, except that each class is now represented by a pair of interface / class and is injected onto the next layer, because we have SOLID.
Morality : concepts require thoughtful application. They can not be used simply everywhere and always, like tools.
Studied generics. Now we have a simple "HelloWorldPrinter" will become "HelloWorldPrinter".
There is no need to use generics when a really necessary form will always be instantiated with the same data types. Parameters are sufficient in most cases.
We studied the Strategy pattern. Now all the if statements will be a strategy .
Why?
Learned to write DSL. Now we have DSL everywhere and for everything.
Well, I don’t even know ...
Moki was discovered. Now we will be chained all along and across.
Oh well ...
Metaprogramming is a great thing, let's use it here and here!
Explain why ...
Extension methods \ concepts \ something else is a good thing, let's use it all around!
And let's not!
Morality: nothing needs to be applied everywhere and always. And the section “Morality” also should not be written in each paragraph.
Vague. Invariably. It’s hard to argue.
Example 1 : Let's create a CMS so that the client can add fields to this form itself.
Result : Customers will never use it. When needed, they will find a developer and puzzle him. Perhaps instead of a full-fledged CMS, it was worth sketching a short instruction on how to quickly add a field.
Example 2 : Let's design a large layer for working with databases, to simplify the "Configurability". We should be able to change the database by editing one config file.
Result: Over 10 years of work, I saw only one project where it was necessary to change one database to another for objective reasons. And, when it came to that, then “editing one config file” did nothing. There was a lot of related work. Incompatibility, gaps in functionality. And one more time a client asked me to transfer HALF of our data models to a new NoSQL database. We had a “magic file” with a database connection, but firstly, only relational, and secondly for all data, not half. Perhaps all these dances with configurability make sense when it comes to something like the Oraklov database, which costs like hiring 20 programmers - it really is convenient to be able to switch to something else if necessary. But for modern databases, all we need is a simple set of vertical DAO classes instead of a wide horizontal ORM layer. There is no single model that successfully combines SQL and NoSQL, so maybe we should really separate them, using one or the other in each case, instead of trying to combine incompatible ones and put up some terrible wrong abstractions.
Example 3 : We built an OAuth authorization system for enterprise clients. For administrators of this system, we were asked to add one more level - administrator authorization through Google OAuth. “Because it should be safe.” If someone breaks into our OAuth authorization, you need at least administrator accounts to be unavailable. Google OAuth is a reliable thing, so there seems to be nothing to mind here.
Result : Anyone who wants to hack this system does not need to break through the OAuth layer at all. You can find the vulnerability in something simpler (there are always such). As a result, all the costs of supporting the two levels of OAuth (and they passed through the entire system) yielded almost no results. It was better to spend this time fixing common vulnerabilities.
Morality: Do not take all of these characteristics with an ending in "-st" as an unchanging given. They have a price - so clearly define the Scenario \ History \ Usage.
It is always cool at the beginning. But in a few years it will already be the Cargo of Heritage.
A couple of examples:
What's so bad:
Moral : Reuse. Borrow the good. Review your decisions.
If you still decide to build your bike - at least do it on the principle of "internal open source" in your company. Explain to people why this is necessary. Show the first sketch. Invite them to use or improve this. And think 10 times, is it worth continuing to build this bike, even if your colleagues do not express their approval.
As soon as you have implemented something in a certain way, everyone else begins to use it in the form in which it is implemented. No one wonders if this is right. As long as your code works, this is “good code.” People begin to apply it even for tasks that it was not originally supposed to solve. It turns out sometimes good, and sometimes very bad. And here you need not to be afraid to take and change your code, improving it for current tasks. Healthy software systems always live, change. Unhealthy - only complemented. If a piece of code has not seen commits for a very long time - something is wrong with it, it “smacks of”. Every piece of code needs to evolve. Here is a great article describing this.
This is how teams work on tasks, but how they should do it, Every Day:
Morality: refactoring is part of any development. No code remains unchanged.
Often we see how nice seemingly good teams or individual coders suddenly give out a frankly weak product. We look at the code base and are surprised - “How so, really it was really created by this team / person, but I considered them / him so smart / smart!”. Quality requires not only skill, but also time. Even smart developers often overestimate their capabilities. And suddenly they find themselves in front of a burning deadline, painfully writing a terrible crutch into the code, giving a chance to somehow meet the deadlines.
Morality : An incorrect estimate of deadlines damages the quality of your project before the first line of code is written.
1. Engineers know better
We engineers consider ourselves the smartest people. Well, since we create different things. And this error often leads to over-engineering. If you have planned and built 100 modules - Business will always ask you for the 101st, which you never thought of. If you gather your strength and solve 1000 problems, they will come to you and lay out 10,000 new ones on the table. You think that everything is under control, but in fact you can’t even imagine which direction the road will lead you tomorrow.
In my 15 years as a programmer, I have never once seen a Business give complete and stable software requirements once and for all. They always change, expand. And this is the nature of the business, not the mistakes of the people who manage it.
Moral : Casino (business) always wins.
2. Reuse of business functionality
When the Business throws us more and more demands (as expected), we sometimes say to ourselves: “Ok, let's try to group and summarize everything that is possible!”.
This is why most MVC systems end with a Big Model or a Fat Controller. But, as we have already found out, business requirements will never stop being added, which means that this is the way to nowhere. In fact, we need to respond to this like this:
Example: once for one of the customers we developed a user profile module. We started with the classic CRUD controller, because this is just a user profile and everything should be simple there. It all ended with the implementation of 13 different scenarios for creating and filling out this profile: registration through a social network, regular, simplified, quick with subsequent editing, looking completely different for subsites, etc. They had very little in common with each other. The scenarios “viewing an order” and “editing an order” are similarly different.
For starters, try dividing business functionality vertically before dividing it horizontally. It even makes the task of dividing the monolith into microservices easier. Sometimes it even becomes monolithic at all, or at your services. Otherwise, it becomes more and more difficult to change parts of your system.
Moral : Prefer isolated actions, avoid combining.
Tip : Take some user-visible part of your program (form \ page \ command). How many context switches does a programmer need to understand all the code associated with it?
3. Generalize everything
(Something in common with the previous paragraph, but sometimes it happens independently of it)
- Need to connect to the database? Let's write a Generic Adapter.
- Make a request? Generalized Request.
- Pass parameter to it? Generalized Parameter.
- Collect a few parameters into something? Generalized Builder.
- Reflect the result in something? Generalized Data Mapper.
- Process user request? Generalized Request.
- Do something? Generalized Contractor.
- etc.
Sometimes engineers bring. Instead of solving business problems, they spend time choosing the right parent class. And the answer is simple.
Software architecture always plays catch-up with business requirements. So even if with the help of some magic you find the perfect abstraction today, its expiration date will immediately come with it - see point No. 1 above - business always wins. So the best measure of the build quality of your architecture is how quickly it can be taken apart. There is a great article on how to write code that will be easy to remove, not easy to change.
Moral : Code duplication is better than incorrect abstraction.
Moreover, code duplication sometimes helps to find that very correct abstraction. Because when you see several parts of a system using the same code in the same way, you can understand what unites them. The quality of Abstraction is determined by the quality of its weakest link. Duplication of the code allows you to look at different situations from different angles and see the boundaries of Abstraction more clearly.
4. Writing wrappers for all external libraries
There is a practice of writing wrappers (wrappers) for each external library used (mainly because of following the style of the main product, sometimes because of the reasons described in the previous two paragraphs). A very popular approach in enterprise programming. Programmers in dynamic languages like Node / Ruby / Python simply add a library and start using it. But the Enterprise Developer cannot go for it - he needs to create a wrapper. And then another wrapper over the wrapper. Perhaps this made some sense in the 2000s, when most of the libraries were a mishmash, and in general there was not much open source code.
But today is the 2016th year. External libraries improved by an order of magnitude. They became just fantastic. Most likely they were written by excellent programmers and tested better than your main code. They have a distinct API. You can embed logging into them. You do not need to spend your time writing wrappers around already good code. In addition, most wrappers are completely meaningless. Their interface is very closely related to the library below, often simply in the form of a reflection of one-to-one functions. If the library changes in the future, most of the code using this wrapper will also have to be changed. Creating an “agnostic” wrapper that can remain unchanged even, for example, with the complete replacement of the library to be wrapped with another, is not a trivial task. Such tasks are sometimes posed with the thought of the potential "configurability" of the general solution,
Morality : wrappers should be exceptions, not the norm.
5. Blind application of code quality metrics
Mindlessly following code quality concepts (like all variables must be declared as “private final”, each class must have an interface) will not make your code AUTOMATICALLY good. Check out, if you haven’t seen it yet, the enterprise version of FizzBuzz or Hello World . Breakthrough code. At the micro level, each class follows SOLID principles and uses great patterns. If you set some code quality analysis tool on this code, it will say that it is a wonderful product, a sight for sore eyes. But if you take a step back, you will immediately see what a nightmare this program is, just printing Fizz Buzz.
Moral : Always take a step back and look at the big picture.
Likewise, automated tools define the test coverage of the code well, but they say absolutely nothing about whether you are testing what you need or not. They can measure performance in some test, but they won’t say how well your program handles data in another case. They all look at the micro level: what does this class \ method \ string do. And only Man looks at the big picture.
Learn a different programming language and try different approaches to solve problems you already know. This will make you a significantly better programmer. Do not get hung up on just one thing.
5.1. Sandwich layers
Let's take some closely related functionality and divide it into 10-20 layers, where each layer has no meaning without the rest. Well, because we want to implement the concept of “Testable code”, or “The principle of shared responsibility”, or call it something else fashionable. In the past, this was done through inheritance. Class A inherits from B, which inherits from C, etc.
Today, people are doing the same thing, except that each class is now represented by a pair of interface / class and is injected onto the next layer, because we have SOLID.
Morality : concepts require thoughtful application. They can not be used simply everywhere and always, like tools.
6. New Syndrome
Studied generics. Now we have a simple "HelloWorldPrinter" will become "HelloWorldPrinter
There is no need to use generics when a really necessary form will always be instantiated with the same data types. Parameters are sufficient in most cases.
We studied the Strategy pattern. Now all the if statements will be a strategy .
Why?
Learned to write DSL. Now we have DSL everywhere and for everything.
Well, I don’t even know ...
Moki was discovered. Now we will be chained all along and across.
Oh well ...
Metaprogramming is a great thing, let's use it here and here!
Explain why ...
Extension methods \ concepts \ something else is a good thing, let's use it all around!
And let's not!
Morality: nothing needs to be applied everywhere and always. And the section “Morality” also should not be written in each paragraph.
7. “X – st”
- Configurability
- Security
- Scalability
- Supportability
- Extensibility
- ...
Vague. Invariably. It’s hard to argue.
Example 1 : Let's create a CMS so that the client can add fields to this form itself.
Result : Customers will never use it. When needed, they will find a developer and puzzle him. Perhaps instead of a full-fledged CMS, it was worth sketching a short instruction on how to quickly add a field.
Example 2 : Let's design a large layer for working with databases, to simplify the "Configurability". We should be able to change the database by editing one config file.
Result: Over 10 years of work, I saw only one project where it was necessary to change one database to another for objective reasons. And, when it came to that, then “editing one config file” did nothing. There was a lot of related work. Incompatibility, gaps in functionality. And one more time a client asked me to transfer HALF of our data models to a new NoSQL database. We had a “magic file” with a database connection, but firstly, only relational, and secondly for all data, not half. Perhaps all these dances with configurability make sense when it comes to something like the Oraklov database, which costs like hiring 20 programmers - it really is convenient to be able to switch to something else if necessary. But for modern databases, all we need is a simple set of vertical DAO classes instead of a wide horizontal ORM layer. There is no single model that successfully combines SQL and NoSQL, so maybe we should really separate them, using one or the other in each case, instead of trying to combine incompatible ones and put up some terrible wrong abstractions.
Example 3 : We built an OAuth authorization system for enterprise clients. For administrators of this system, we were asked to add one more level - administrator authorization through Google OAuth. “Because it should be safe.” If someone breaks into our OAuth authorization, you need at least administrator accounts to be unavailable. Google OAuth is a reliable thing, so there seems to be nothing to mind here.
Result : Anyone who wants to hack this system does not need to break through the OAuth layer at all. You can find the vulnerability in something simpler (there are always such). As a result, all the costs of supporting the two levels of OAuth (and they passed through the entire system) yielded almost no results. It was better to spend this time fixing common vulnerabilities.
Morality: Do not take all of these characteristics with an ending in "-st" as an unchanging given. They have a price - so clearly define the Scenario \ History \ Usage.
8. Bicycle construction
It is always cool at the beginning. But in a few years it will already be the Cargo of Heritage.
A couple of examples:
- Own libraries (HTTP, mini-ORM, caching, configuration)
- Custom frameworks (CMS, event processing, multithreading, background tasks)
- Own tools (assembly system, deployment system)
What's so bad:
- Trivial, it would seem, tasks actually require decent knowledge and deep immersion in the subject area. Some kind of “process launcher” requires an understanding of process management in the OS, the work of daemons, an input / output system, and a whole bunch more. CMS is not just something that substitutes data in templates - there are dependencies, checks, wizards, generalizations, etc. The simplest-looking library may have non-trivial functionality.
- All this also needs to be maintained, updated.
- If you put the code in the public domain, everyone will not care. Well, maybe, except for those who worked on this code before.
- People who understand this code will leave sooner or later. And besides them, no one in the world understands this implementation.
- Implementing existing libraries and frameworks into the project, sharpening them to your needs requires time right now. But inventing your own bicycles requires much more time in the long run.
Moral : Reuse. Borrow the good. Review your decisions.
If you still decide to build your bike - at least do it on the principle of "internal open source" in your company. Explain to people why this is necessary. Show the first sketch. Invite them to use or improve this. And think 10 times, is it worth continuing to build this bike, even if your colleagues do not express their approval.
9. Develop your code
As soon as you have implemented something in a certain way, everyone else begins to use it in the form in which it is implemented. No one wonders if this is right. As long as your code works, this is “good code.” People begin to apply it even for tasks that it was not originally supposed to solve. It turns out sometimes good, and sometimes very bad. And here you need not to be afraid to take and change your code, improving it for current tasks. Healthy software systems always live, change. Unhealthy - only complemented. If a piece of code has not seen commits for a very long time - something is wrong with it, it “smacks of”. Every piece of code needs to evolve. Here is a great article describing this.
This is how teams work on tasks, but how they should do it, Every Day:
Morality: refactoring is part of any development. No code remains unchanged.
10. Invalid deadlines
Often we see how nice seemingly good teams or individual coders suddenly give out a frankly weak product. We look at the code base and are surprised - “How so, really it was really created by this team / person, but I considered them / him so smart / smart!”. Quality requires not only skill, but also time. Even smart developers often overestimate their capabilities. And suddenly they find themselves in front of a burning deadline, painfully writing a terrible crutch into the code, giving a chance to somehow meet the deadlines.
Morality : An incorrect estimate of deadlines damages the quality of your project before the first line of code is written.