Understand open source

Open Source makes me feel like a masochist.
I love him, but often working with him is a pain.
(c) mine

Once upon a time - maybe 10 years ago - it was fashionable to predict the end of open sors. The Internet was full of headlines like “Why Open Source will die soon” or “Why Open Source will never beat paid software.” These are not exact headings. Rather, the impression that has been preserved since then.

Now we see that open sors blooms and smells. Sometimes it smells good, sometimes it doesn't. But in general, it is still gaining popularity and even somehow evolving. Actually, we’ll talk about this: about what Open Source is now, and where it will move on. I would like to see your opinions on this in the comments.

I’ll immediately warn you that I’m not an analytical agency. Therefore, I will not provide large-scale statistics with far-reaching conclusions built on it. Rather, I will confine myself to “far-reaching conclusions” based on my work with open source, on communication with people who also do this, and on monitoring the activities of various companies in this area.

I have planned three posts that I will post if the topic seems interesting:

1. This post, where we talk about how to learn to understand Open Source
2. Models of using Open Source to achieve business goals
3. In the third post, I will propose a loose classification of open source by its suitability. By how convenient, efficient and useful it is to work with various third-party projects.

What is open source, I first found out, probably, in the late nineties. The first time I assembled the Linux kernel from the source code in the year 2001. Until 2008, open source I was only interested in as a “free” toolkit for home and work. I quote the word “free” because for a long time I have not considered the combination of “Open Source” and “paid software” to be opposite in meaning. But more on that later. In 2008, for the first time, I began to reluctantly observe trends in open source from within the company I worked for then. But he really started working with him less than two years ago. But I already managed to cool off a dozen of about a dozen different projects.

All this time, myths related to open source have been debunked in my brain. And I even suspect that the real story of open source developed just in such a way as to dispel myths that were once true. That's only in reality they collapsed much earlier than in my head.

By the way, a small questionnaire: how do you use open source? (See at the very bottom of the post. Habr is not able to insert questionnaires into the text)

At first, I looked at open source software very romantically. In fact, I believed in the legends that were widely circulated at one time about “red-eyed enthusiasts” who wrote code in the night with the sole purpose of making the world a better place. Of course, there were such people. Even in my student dormitory there were guys who picked the Linux kernel and GNU applications. And some of them even injected their code into the corresponding repositories. I suspect that even now in dormitories of technical universities, people at night are engaged not only in sex and gaming. It’s even interesting what they are picking right now. If 15 years ago the answer was obvious - Linux - now I am almost sure that it is not Linux. Obviously, enthusiasts with red eyes still exist today. But their contribution to Open Source is negligible.

The first myth I had to part with was the idea that they don't make money on open source software. It was very easy to say goodbye to this idea, because there was no need to go far for examples. Take at least corporate versions of Linux distributions. For example, Red Hat Enterprise Linux (in common people - RHEL). Commercial Linux distributions are an iron example of what open source can earn. You will not dismiss him as an argument like “If someone is trying to make money, this does not mean that the idea is popular and that you can earn a lot.” Almost every major Western company pays for such distributions. And how much you can earn from this is easy to understand from the financial statements of Red Hat or from a brief summary of its financial metrics .

Free open source is the second myth that I said goodbye to. In a good way, this myth should be more or less equivalent to the first with some stretch. However, in the minds of people, these two delusions lived separately. My case was no exception. It is funny that in the modern world, the first myth, apparently, has already expired. But the second is still flourishing. I think there are several reasons for this:

1. people think that only money can pay. Although if a person spends the night trying to get the "free" software to work, he pays his own time. (Which, as you know, is still almost money). Endless fuss with graphics libraries is the reason why I stopped using Linux GUIs for home purposes (although now the situation with them has probably already improved);
2. the real payer is overlooked.Such a situation is often found on Habré. “I will never buy this expensive integrated boxed solution. I’d better buy cheap iron for a penny and put open sourcing on it. Everything will work in exactly the same way and even better! ”Here the interesting point is that the real thing is that“ everything will work ”, it is not the commentator who pays, but his employer. It is he - i.e. tenant - paid the craftsman to make everything work. And he could fire the craftsman and approach the secretary with the money released: “Please order this black box. And when it comes, please plug it into a power outlet. You don’t have to do anything else. ” Of course, I'm exaggerating a bit here. And even more than that: for people to start costing less than the solutions delivered with the service, the business needs to achieve certain sizes. But here we come close to the following reason;
3. size matters. Everyday “everyday” experience deceptively hints to us that open source is free. You wrote, for example, Hello World in C, compiled it with the open GCC compiler, and everything is fine with you.But what if you have several million lines of code? And if you want to include non-trivial optimizations in the compiler? With a very high probability, you will run into trouble. By the way, you will have them not only with the "free" compiler. The problems will be regardless of which compiler you use - open source or not. You will have to pay anyway. As soon as we start to do something big and - which is also important - non-trivial, we come to the conclusion that we have to pay for tools. In my previous place of work, even a relatively small team of 20 developers once in a couple of months encountered errors in the Linux kernel. And every time new servers were bought, it was necessary to re-integrate the same Linux with them. This is an illustration of the fact that if your business is large-scale, you need support. You can pay for it by buying a “solution” with support in the kit, or you can put a free distribution kit and buy support separately. However, you will lay out the money.
   
   Here are a few more reasons to pay for Open Source:
   
   1. poor documentation
   2. incompatibility with hardware or software platform
   3. non-trivial user interface for which you need to train staff
   
   Now the third myth. This time, it is as if enthusiasts are working on open source. This error is connected with the first two and was chronologically dispelled simultaneously with them. If you take a closer look at the popular Open Source projects, you will notice that the most interesting and high-quality open source is done by large commercial companies. For example, Java is the brainchild of Sun Microsystems. The now popular LLVM compiler was conceived by Apple, and later Intel and Google joined in its development. The indispensable in the modern world Hadoop is almost completely created by Yahoo! ( Between 2006 and 2011, Yahoo! poured about 90 thousand lines of code into Hadoop, the closest "pursuer" - about 10 thousand ).
   
   
   
   
   
   By the way, about Yahoo! .. About 15 years ago, almost every Internet user periodically used the search engine of this company. Check how many people know Yahoo! now. At the end of the post on this topic there are questionnaires.
   
   
   
   
   
   Above, I mentioned only the main misconceptions that exist or existed regarding Open Source. There are still a bunch of small ones. But discussing them is no longer so interesting. It is much more interesting to talk about how to understand the trends that drive the development of modern open source. If you learn to analyze the largest events in the world of open source software, then you do not need prejudices to understand the big picture.
   
   There can be no universal recipes here. I can only share what I consider useful for myself. I will be glad to hear in the comments about what you rely on.
   
   So, how to learn to understand Open Source?
   
   Recipe 1. Open Source is almost always about money.If you see incomprehensible but powerful “movements” in the open source community, try to understand who is trying to make money on it and how. Let's take an example.
   
   Less than a year ago, Microsoft announced the release of .NET Core in Open Source. Question: why?
   
   
   
   
   
   If you try to understand this using a search engine, then in Google’s top you can link to this . It follows from the article that the goal is to bury Java. Another question pops up on Quora , where the most popular answer says the same.
   
   Well, let's say that the goal is really that. But how is Microsoft going to capitalize on this? We know that Microsoft is not a charity. While it remains unclear how the company wants to make money, we can assume that we have no answer.
   
   We are looking further. We find the official answer of Microsoft . By the way, I managed to find it only thanks to Yandex, although I used the same queries as in Google.
   
   What does Microsoft itself tell us?
   
   
   
   
   
   Read the article. There are many correct and beautiful words about what the client will receive from the new initiative. But not a word about what the company itself wants to get. Actually, this could not be there. Companies always focus on the customer. “Everything for the client, damn it!” And this is absolutely correct.
   
   I could not find a suitable explanation on the Internet (perhaps you are more fortunate in this matter). So, we begin to think independently.
   
   Microsoft itself tells us (see the picture above) that .NET went open sourcing for the sake of cross-platform. Wonderful! Microsoft will not lie to us, it just will not give out the whole truth :). We look at other company initiatives in this area. We notice that Microsoft is a great apologist for HTML5. I will not give specific links here, because The company's activities in this area are multifaceted. Any search engine will easily show everything.
   
   
   
   
   
   We look further. We see that the company recently announced - through one of its subsidiaries - the JUniversal project . This is a tool for translating Java code to the .NET platform.
   
   
   
   
   
   So, we found that the company is very interested in cross-platform applications. And not only on .NET. But why then did Microsoft not rush to develop Java?
   
   Again, remember that Microsoft is not for charity. Java is controlled by wayward company Oracle. Putting yourself at the whim of Oracle is a big risk. Therefore, Microsoft is concentrating either on the tools that it controls itself, or on those technologies that no one individually controls. Those. it seems like the company is really not averse to eating a piece from Java. But that is not the goal itself. What then is the goal?
   
   The answer often flashes on the Internet pages that are dedicated to the cross-platform efforts of the office. If you repeat the search that I briefly described above, you will see that the goal is to make the applications that developers write for Android and iOS also run on Windows / Windows Phone. Bingo! Add the last brick: Microsoft does not have a rich ecosystem around its mobile platform. But Google and Apple have such an ecosystem. So Microsoft wants to take advantage of an alien ecosystem to entice users of two mobile giants. "Guys! All your favorite applications work on our system! Ayda to us. Our platform is more glamorous ”(for example).
   
   
   
   
   
   Did I make the right conclusion? I do not know. But at least this is a hypothesis that can be checked by observing the further actions of the company.
   
   Such logical constructions when analyzing trends in Open Source have to be done often. Because soft giants will never come and say in plain text: "You know, we are going to exploit a strange ecosystem here."
   
   A long example turned out ... I hope for the good. We are going further.
   
   Recipe 2. Thinking of where Open Source might go, look for analogies. Truly new ideas appear very rarely in the world. The idea that open sorsu in the broad sense for many hundreds of years is sometimes useful to me. What I mean?
   
   Legally, open source is a publication. About how thin. The code is also subject to copyright. The use of fiction or scientific publications may be licensed. You can do the same with code. The technologies described in scientific papers can be patented. The algorithms implemented in the code, too (although not in all countries; and often - with a completely different sauce).
   
   How can you use similar analogies? As always, I will give an example.
   
   Let's take something not the most trivial. Let's talk about the role of the state.
   
   Publications are beneficial to the state. Thanks to them, private ideas become public knowledge, which is a serious engine of the economy. Therefore, the state makes every effort to “lure” these ideas out of the population. So, for example, one of the main crutches of modern capitalism was created - the patent system. Patents make the idea public, but protected from extraneous attacks. Its further use is regulated by licenses.
   
   So, in the world of publications, the state plays a huge role. And many publications interesting for the development of the economy are born in the academic environment, which is supported by the same state. What hypothesis could be advanced now?
   
   That's right: the state plays a huge role in the development of Open Source. And the academic environment too. This refers, first of all, to the American state. Russian for the development of Open Source has so far done little.
   
   Before testing the hypothesis, let's look a little deeper at the situation. How are technical ideas born in an academic setting commercialized?
   
   So that scientists do not “clamp down” their ideas, the state allows them to be patented and even gives priority to commercialization (with some reservations, which are not significant here).
   
   As a result of scientific work, technology is created that is protected by patents and lined with licenses. Further, scientists - provided that the university allows it - establish a startup and begin to bring technology to the product. If everything goes well, they make money from it.
   
   An important point here is that scientists start a business with already finished technology in their hands, created by them and paid for by the state.
   
   Now what do we see regarding our hypothesis? And the fact that a huge amount of open source niche software was really born within the walls of research organizations. An example from personal practice. This summer I attended the ISC'15 (International Supercomputing Conference). Almost all the software presented at the conference is open and originates in academic circles. At first it was developed with state money, and then the developers, leaving the laboratories with ready-made technologies, created commercial offices around them. This includes, for example, all the most popular MPI implementations and task schedulers .
   
   I constantly encounter open source that has come out of the academic environment. Often these are very serious projects. Take at least a tool for visualizing VisIt data . 1.5 million lines of code. Developed by a very large American laboratory LLNL. It can be used both on its own and as a library for Python, C ++, Java. By the way, I recommend it.
   
   
   
   
   
   
   
   
   
   Here is another example of the role of government money. "Shaggy" list of 2003. World famous Open Source projects sponsored by the state.
   
   So, drawing an analogy with other types of publications, we easily came to the conclusion that the state is a serious driver of open source.
   
   Perhaps this post is read by intellectual property specialists. Maybe one of them knows where you can get statistics on the amount of money that the US invests in Open Source? And the number of commercial companies that have left the academic environment on the basis of Open Source?
   
   However, one more recipe remained.
   
   Recipe 3. Periodically refuse recipes.Or look for new ones. Or do not use at all. Open Source is evolving rapidly. What seemed fantastic a couple of years ago is already a reality today.
   
   This “recipe” cannot be illustrated by example due to the very essence of the recipe. But I’ll say something anyway :)
   
   Today I can “google” or “index” the Python library for almost any application. Those. at some point, a truly convenient tool appeared in the world that gave an additional impetus to the development of the Open Source community. It became very easy to develop elegant, small but effective tools. (Although with a limited scope). Each of them individually is almost imperceptible and, most likely, will never enter any product. But all together they create the basis for a new type of social network, or something ... People post their snippets instead of photos of legs and food. A significant role in the development of this phenomenon also belongs to GitHub.
   
   Convenient tools led to a small Open Source communism, or something ... People post their work on the network only because it will be nice if someone else uses it.
   
   That's all for now. In the next post we’ll talk about what goals individuals and companies go to Open Source.
   
   Continuation: “Understand Open Source: usage patterns”
   
   Two promised questionnaires: