The essence of software development
- Need to make 500 holes in the wall, so I designed an automatic drill. It uses elegant precision gears to continuously adjust speed and torque as needed.
“Great, she has perfect weight.” We will load 500 such drills into the gun, which we made, and shoot at the wall.

I have been programming for 15 years. But lately, when developing, it is not customary to think about efficiency, simplicity and perfection: to the extent that it makes me sad during my career and for the IT industry as a whole.

For example, modern cars work, say, 98% of what the current engine design physically allows. Modern architecture uses a precisely calculated amount of material to perform its function and remain safe in these conditions. All planes converged to the optimal size / shape / load and basically look the same.

Only in software is considered normal if the program works at a level of 1% or even 0.01% of the possible performance. No one seems to have any objections. People even take pride in how inefficient the program works, like “why bother, computers are fast enough”:

@tveastman : I run a Python program every day, it runs in 1.5 seconds. I spent six hours and rewrote it on Rust, now it runs in 0.06 seconds. This acceleration means that my time will pay off in 41 years, 24 days :-)

Probably, you heard such a mantra: “The time of a programmer is more precious than the time of a computer.” This means that we spend computer time on an unprecedented scale. Would you buy a car with a consumption of 100 liters per 100 kilometers? What about 1000 liters? With computers this happens all the time.

Everything is unbearably slow

Look around: laptops are thousands of times more powerful than those that brought man to the moon. However, every second site cannot provide smooth scrolling of 60 FPS pages on the latest top MacBook Pro. I can comfortably play games, watch 4K videos, but not scroll through web pages! This is normal?

The Google Inbox email application in Chrome browser from the same Google, takes 13 seconds to open a medium-sized letter :


It also animates empty white forms instead of showing their contents, because this is the only way to animate something on a web page with decent performance. No, not 60 FPS, but rather "as fast as possible on this page." I look forward to what the web community offers when 120 Hz displays become mainstream. They barely cope with 60 Hz.

Windows 10 update takes 30 minutes . What can be done for so long? This time is enough to completely format my SSD-drive, download a fresh build and install it approximately 5 times in a row.

Pavel Fatin : Typing in the editor is a relatively simple process, so even 286 could provide a fairly smooth typing process.

In modern text editors, the delay in typing is more than in 42-year-old Emacs. Text editors! What could be easier? For each keystroke, you only need to update the tiny rectangular area on the screen, and modern text editors cannot do it in 16 ms. And this is a lot of time. LOT. A 3D game fills the screen with hundreds of thousands (!!!) of polygons for the same 16 ms, and also processes input, recounts the world and dynamically loads / unloads resources. How so?

The trend is that the software does not become faster and more functional. We get faster hardware, on which software with the same functions is turned more slowly than before. Everything works much slower than the maximum speed. Have you ever wondered why your phone boots from 30 to 60 seconds? Why can't it boot, say, in one second? There are no physical limitations here. Personally, I would like this. I want the developers to reach the limit, using every bit for performance.

Everything is HUGE

And this is blowing up. Web applications can open ten times faster if you just block ads. Google is begging everyone to stop braking with the help of the AMP initiative - a technical solution that doesn’t need any technology, just a little common sense. If you remove bloat, the Internet will work at a crazy speed. Is it really hard to understand?

The Android system without applications takes up almost 6 GB . Just think for a second how indecent this huge number is. What is it, movies in HD quality? I think basically the code: kernel, drivers. Still some resources, of course, but they cannot be so big. How many drivers do you need for the phone?



Windows 95 occupied 30 MB. Today we have web pages harder than this OS! Windows 10 is already 4 GB, that is 133 times more. But is she 133 times better? I mean, functionally they are almost the same. Yes, we got Cortana, but I doubt that it weighs 3970 MB. But this is Windows 10, should Android really be another 1.5 times as big?

The Google keyboard application, as if nothing had happened, eats up 150 MB. This program draws 30 keys on the screen - is it really five times harder than the entire Windows 95? The Google app is basically just a package for Google Web Search, it takes 350 MB! Google Play services that I don’t use (I don’t buy books, music or videos there) - 300 MB, which just sit here and which cannot be removed.



After installing all the necessary applications (social networks, chat rooms, maps, taxis, banks, etc.), only 1 gigabyte for photos remained on the phone. And it is generally without games and music! Remember the times when the OS, applications and all your data were placed on a floppy disk?

Your note-taking program is most likely written in Electron and, therefore, comes with a driver for the Xbox 360 controller , can display 3D graphics, play audio and take photos using a webcam.



Simple text chat has always been famous for speed and low memory consumption. So Slack is an example of a very resource-intensive application. I mean that chat and text editor are the most basic things, they should consume the least resources. Welcome to the year 2018.

You can say that they at least work. But increasing the size doesn’t mean improvement. This means that someone has lost control. We no longer know what is happening. Increasing size is an increase in complexity, a decrease in performance and reliability. This is not normal and should not be considered normal. On the bloated size you need to immediately pay attention - and stay away from them.

All rotting

The 16 GB Android phone was beautiful three years ago. Today, under Android 8.1, it barely works, because every application has doubled for no apparent reason. No additional features. They do not become faster and the appearance has not changed. Are they just ... bloated?

iPhone 4s came out with iOS 5, but it can barely work with iOS 9. And this is not because iOS 9 is much better - basically, the system has not changed. But new hardware is faster, so they made software slower. Do not worry - you got exciting new features, for example ... the work of the same applications with the same speed! I do not know.

iOS 11 has stopped supporting 32-bit applications. This means that if the developer is not ready to return and update the application, most likely you will not see this excellent program again.

@jckarter : The DOS program can be made to work without changes on almost any computer made after the 80s. A javascript app may stop working due to tomorrow's Chrome update.

Today's web pages will not work in any browser in 10 years (and maybe earlier).

"We need to run as fast as possible, just to stay in the same place." But the point? I can constantly buy new phones and laptops, like everyone else, but do it only in order to be able to run all the same applications that have only become slower?

I think that we can and should correct the situation. Now everyone is developing programs for today, occasionally for tomorrow. But it would be nice to do things that work a little longer.

Worse means better

Now nobody understands anything. And do not want to understand. We just release half-hearted nonsense, hope for the best and call it “common sense for a startup.”

Web pages are asked to update if something went wrong. Who has time to find the cause of the problem?



Any web application produces a constant stream of “random” JS errors, even on compatible browsers.

The entire web / SQL database architecture is built on the premise (even hope) that no one will change the data while you look at the open web page.

Most collaboration applications have done “as best they could”, there are a lot of typical scenarios when they lose data. Seen the dialogue "Which version to keep?"



And no, in my world there is not a normal application that says: "I will destroy part of your work, just choose which one."

Linux intentionally kills random processes. And yet it is the most popular server OS.

I have every device fails regularly anyway. From time to time, the Dell monitor needs a hardware reboot, because it has software. AirDrop? You are lucky if he finds the device, otherwise what to do? Bluetooth? Specifications are so complex that devices will not communicate with each other , and periodic reboots are the best option .



And I don’t even mention the Internet of Things . It is so beyond reason that even nothing to add.

I want to be proud of my work. I want to do working, stable things. To do this, you need to understand exactly what we are developing, inside and outside, and this cannot be done in bloated, overly complicated systems.

In programming, the same chaos

It seems that no one else is interested in high-quality, fast, efficient, durable, solid solutions. Even if effective solutions have been known for a long time, we still struggle with the same problems: package management, build systems, compilers, language design, IDE.

Assembly systems are inherently unreliable and periodically require complete cleaning, although they have all the information for invalidation. Nothing prevents to make the assembly process reliable, predictable and 100% reproducible. Nobody just thinks it's important. NPM has been in the "sometimes running" state for many years.

@przemyslawdabek: It seems to be rm-rf node_modulesan integral part of the workflow in Node.js / JavaScript projects.

And the build time? No one considers it a problem that the compiler is running for minutes or even hours. But what about “time programmer more expensive”? Almost all compilers, pre- and postprocessors significantly, sometimes catastrophically increase the build time, without providing proportionally significant advantages. You expect programmers to make mostly rational decisions, but sometimes they do the exact opposite. For example, choosing Hadoop even if it is slower than performing the same task on the same desktop computer . Machine learning and AI have dropped software to fortune telling at the thick of the coffee at a time when most computers were not even sufficiently reliable.

@rakhim : When an application or service says "under AI control" or "based on machine learning," I read it as "unreliable, unpredictable behavior that defies explanation." I stay away from “AI” because I want the opposite of computers: reliability, predictability and logic.

We stuffed virtual machines in Linux, and then stuffed Docker into virtual machines, simply because no one could deal with the mess that most programs, languages, and their environments make. We cover the shit with blankets so as not to take it away. For example, the “single binary” remains a HUGE Go advantage. No mess == success. Python's environment is so polluted that my laptop was declared an environmental disaster zone. Note. The Python Environmental Protection Agency wanted to fill it with cement and bury it with a picture at the entrance - a warning for future civilizations about the dangers of using sudo to install random packages





And addictions? People thoughtlessly put over-complicated “full packages” for the simplest of problems, without thinking about the consequences. From these dependencies grow new. Ultimately, you get a tree, which is something between a horror movie (huge and full of conflicts) and comedy (there is no reason why we added these packages here, but here they are ): Programs cannot work for several years without rebooting. Sometimes even a few days is too much . Occasional glitches happen, and no one knows why.





Worse, no one has time to stop and find out what happened. Why bother if there is always another way. Raise the new AWS instance. Restart the process. Delete and restore the database. Write a script that will restart your broken application every 20 minutes. Include the same resources several times: tyap-lyap - and in production . Move fast, do not waste time on correcting errors.

This is not an engineering job. This is just lazy programming. Engineering requires a deep understanding of the performance, structure, and limitations of what you create. Sculpting hack from low-quality material is quite the opposite. To grow, we must understand what we are doing and why.

We are stuck

So all this is just a bunch of barely working code added over previously written barely working code. It continues to grow in size and complexity, reducing the chances of change.

To have a healthy ecosystem, you must return. It is sometimes necessary to throw out trash and replace it with better alternatives.



But who has time for it? New OS kernels did not come out how many, 25 years? It has now become too complicated to just take and rewrite. Browsers have accumulated so many borderline situations and historical precedents that no one dares to write the engine from scratch.

Today's definition of progress - or toss fuel:

@sahrizv : 2014 - you need to implement microservices to solve problems with monoliths.
2016 - you need to implement Docker to solve problems with microservices.
2018 - you need to implement Kubernetes to solve problems with Docker.

or reinvent the wheel:

@ dr_c0d3 : 2000: Write 100 lines of XML to “declaratively” configure servlets and EJBs.
2018: write 100 lines of YAML in order to “declaratively” set up microservices.
There were at least schemas in XML ...

We are stuck, and no one will save us.

Business doesn't care

Users too. They learned to accept what we do. We (engineers) say that each Android application takes 350 MB? Well, they will live with it. We say that we can not provide a smooth scrolling? Okay, they get used to the phone, which slows down. We say: “If it doesn't work, reboot”? They will reboot. After all, they have no choice.

There is no competition either. All build the same slow, bloated, unreliable products. A random leap forward in quality gives a competitive advantage (iPhone / iOS versus other smartphones, Chrome versus other browsers) and forces everyone to regroup, but not for long.

Our mission as engineers is to show the world the amazing capabilities of modern computers in terms of performance, reliability, quality and usability. If we care, people will reach out. And no one except us will show them that this is possible. Unless we don't give a damn.

Not so bad

Sometimes rays of hope shine through the overcast sky.

The work of Martin Thompson ( LMAX Disruptor , SBE , Aeron ) is impressive, refreshingly simple and effective.

Editor Xi Rafa Leviena seems to be built on the right principles.

Jonathan Blow has developed a compiling language for his game that compiles 500,000 lines per second on a laptop. This is a cold compilation, no intermediate caching, no incremental builds.

No need to be a genius to write fast programs. There is no magic here. The only thing that is required is not to build software on the basis of a huge pile of shit, which is supplied by modern tools.

Better World Manifesto

I want to see progress. I want a change. That the modern software was improved, but did not stand still. I do not want to reinvent the same thing, each time releasing an increasingly slow and bloated product. I want to believe in something - in a worthy goal, in a future that is better than what we have today and for a community of engineers who share this vision to emerge.

What we have today is not progress. We barely reach business goals with these bad tools. We are stuck in a local optimum, and no one wants to move. It is not even a good place, it is bloated and inefficient. We just somehow got used to it.

Therefore, I want to say: the current situation is complete shit. As engineers, we can and should and will do better. We can have better tools, we can create better applications, faster, more predictable, more reliable, use less resources (less orders of magnitude!). We must deeply understand what we are doing and why. We must produce products reliably, predictably, with the highest quality. We can and should be proud of our work. Not just "considering what we had ..." - no reservations!

I hope I'm not alone. I hope that there are people who want the same. I’ll be happy if we at least start talking about how absurdly absurd the current situation in the software industry is. And then, perhaps, we will figure out how to get out of it.