“Linux kernel development is a club of interest”

    Our architect of the server virtualization department Pavel Emelyanov gave an interview to the System Administrator magazine. We decided to publish his interview here, in which he talked about the CRIU project, how the development team works with the Linux community and Linus Torvalds, and about the changes that may occur in the field of virtualization in the coming years.
    image



    Please tell us what you do in Parallels?
    I am working on a Parallels main server product called Cloud Server. This is a Linux distribution specially prepared to run virtual machines and containers, which is neatly integrated with a large number of additional applications from Parallels - cluster management, distributed storage, web panels and the like. This product is multicomponent, its components are sometimes very complex systems in themselves. I am engaged in the design of many of these systems, mainly related to the core, their internal structure, logic of work and interaction with each other.
    In addition, I will organize the process of interaction between the Parallels nuclear team and the Linux kernel community. The fact is that Parallels is actively involved in the development of container virtualization subsystems in Linux, almost all of our specialists do something for the Linus Torvalds kernel. I follow this activity and try to direct it in the direction necessary for the company.
    Today, my favorite pastime is developing the CRIU project, which came out of our work with the Linux community. This is an application that can remove the status of processes running on Linux and restore them in another place or at another time from the received data.

    How did the CRIU project come about? What exactly prompted its creation?
    The project came about as a solution for integrating Parallels container code into the Linux kernel. And integration itself is also an interesting story that I have already mentioned.
    A few years after the launch of the OpenVZ project, Parallels realized that it would be very difficult to support all the changes that were made in the kernel on their own.
    We decided to start porting our container kernel code to the Linux kernel and send these changes to the community with a request to accept them. I was just a developer then, and the choice of “who will send” fell on me. I started porting our code to another kernel, sending these changes and talking to people from the community about the adoption of new functionality.
    For a couple of years, we managed to integrate about half of all the nuclear functionality that Parallels had, but we did not take the code that we were involved in the “live migration” of containers into the Linux kernel. And not only we tried, other companies wanted similar functionality, but the community didn’t come to a single solution. Then I decided to try to make the required functionality not in the kernel, but as an application, expanding the kernel as needed.
    The first prototype was written in about three days at home while I was on sick leave. I sent him “to see” to the community. People looked and ... decided that they took those small changes in the kernel that I needed (these changes finally settled in the Linux kernel after a couple of months).
    After that, the project began to develop more actively, since I became confident that they would definitely take the additional changes that I would need.

    By whom and where is CRIU applied?
    As for applicability, the project is now at a “transitional age." On the one hand, different companies showed great interest in the project. For example, people from Samsung, Huawei and even recently from Google itself have already sent me corrections and additions. And since they sent it, it means that they are trying to do something with it. But, on the other hand, active use has not yet been observed, and it is clear why.
    The project is quite complex, and we have not had time to stabilize it, so it is not yet possible to use it seriously.

    How many developers are working on the CRIU project?
    Complex issue. So far, three, including me, are actively and regularly doing something with the project. There are still a dozen people who "help", occasionally sending messages about bugs, sometimes with corrections, asking if the project can do this and that, thus giving us new ideas. And also this side - we periodically change something in the core and send these changes to the community, thus implicitly involving people from it in the process.

    What can you expect new in CRIU in upcoming releases? What functionality is planned to be added to the project in the future?
    Oh, in the near future the most important event that will happen with CRIU is that all the necessary nuclear support will come out with a 3.11 kernel. If someone wants to use CRIU "for the full program", he will not have to compile and install a custom kernel. In addition, according to secret intelligence reports, the next version of the Red Hat distribution called RHEL7 will be based on 3.11, that is, it is likely that CRIU will immediately work on one of the most popular Linux distributions.
    And what will happen to the project next is very interesting for me. The fact is that the capabilities that CRIU is able to provide go beyond the interests of Parallels. But who will implement them and when? I have plans to create a community around CRIU, similar to what is built around the Linux kernel so that the project does not develop with the efforts of several people from Parallels, but with a more diverse and large group of developers.

    You said that all the necessary support for CRIU will be in the kernel 3.11. What is missing in the kernel today for full CRIU support?
    If we take the most recent core 3.10 for today, then there are no two things.
    The first, most important, is a subsystem that allows CRIU to track which sections of memory the process is changing so that when the state is removed again, it does not copy all its memory, but only copy what has changed. This is not a necessary component for CRIU, but it can significantly accelerate the process, for example, live migration. Sometimes this is very important, because without this optimization it is possible, for example, to migrate a process so long that its network connections will break.
    The second part is slightly smaller, but in a sense it is much more important than the first. This is a small extension to the process debugging subsystem, which allows you to control how processes process signals. Without it, it is impossible to remove the state from processes that are in a stopped state.
    It sounds scary, but, fortunately, such situations are relatively rare, so CRIU still does without it.

    What Linux kernel projects are you working on besides CRIU? Which non-Linux kernel projects are you involved in?
    Seriously almost no and no. What I have to do as part of OpenVZ, CRIU, and our integration with the Linux kernel is more than enough.

    You mentioned the OpenVZ project. Tell us what this project is. What is the degree of your participation in it?
    From the software point of view, this is the kernel from Parallels + of several utilities for its configuration, with which you can create and manage Linux containers.
    In addition to software, it is also such a (already) brand with which Parallels makes itself known in the Open Source world. Around the project, a medium-sized community has even grown.
    The degree of my participation in it, apparently, is rather big. Soon after the project was born, I was “promoted” from a simple developer to the leader of a team of nuclear scientists. And since the main thing that was valuable in OpenVZ at that time was the core, I de facto became something of a technical leader for the project. A year later, the mentioned campaign to integrate OpenVZ code into the Linux kernel started, which was also conducted under the OpenVZ flag and initially fell on me personally.
    Now the kernel is not the main value of the project (since we have already integrated a lot into the Linux kernel, and, for example, on Fedora 19 you can make containers without our kernel). So, together with the project manager (Kir Kolyshkin), I’m making up how to move the project further without “leaving” the core alone.
    One of these areas is, for example, our CRIU, which is positioned as a subproject of OpenVZ.

    What are the strengths and weaknesses of the OpenVZ project?
    The strong point is that this project on time compared to competitors realized the need for integration with other projects and started from the main thing - from the core.
    The weak side is that it is very closely associated with the closed commercial product of Parallels, and due to inevitable conflicts of interest, decisions are not always made in favor of OpenVZ. But Parallels is well aware of this and is trying to rectify the situation.

    What difficulties arose in the process of integrating OpenVZ code into the Linux kernel? What version of the kernel did the integration begin with?
    I already forgot when we sent the first patches. I don’t even remember which subsystem we started with. I only remember that in the 2.6.18 kernel, on the basis of which we made a stable branch, we already had part of the code integrated.
    In my opinion, these were the namespaces PID and SysVIPC. By 2.6.32, on which we had the next stable version, network virtualization (net namespaces) and something else were integrated.
    Difficulties arose in the fact that I had to "on the fly" to master a completely different way of development. We roughly imagined what the process of integrating patches into the kernel looks like. But, when, in response to my first set of patches, people began to actively discuss how this can be altered from scratch, or suggest, before doing what I did, first rewriting a good piece of the process control subsystem, I was even confused.

    How does the OpenVZ project relate to the LXC?
    And this is generally a wonderful question. The OpenVZ manager has long and hopelessly wanted the answer to it to be printed in huge letters and hanged in some insanely popular place where everyone-all-all could read it!
    First of all, what is the LXC? LXC is, firstly, a set of kernel components that allow you to isolate processes from each other, and secondly, it is a utility that forces these components to interact in the right way, creating a container. OpenVZ is almost the same: nuclear components, which are implemented differently, and utilities that these components configure to create a container too, but more secure and more functional.
    So the answer to the question is this: the Parallels nuclear team has created most of the LXC nuclear functionality. All the functionality that appears in the main core (i.e., de facto in the LXC) necessarily begins to be used by the OpenVZ project. As soon as we make and release our own kernel, based on a newer version of the Linux kernel, we throw out our implementation of the subsystem and replace it with a similar one, which we (usually we) wrote for Linux. That is, the further, the more parts of our kernel code change the brand from OpenVZ to LXC.

    What functionality is planned to be implemented in OpenVZ in the near future? Which in the future? Are there any other subprojects planned for OpenVZ that are related or not related to the Linux kernel?
    Now we are working on a few things. This is an almost completed virtual disk driver for containers, the so-called loop device for those who work with Linux. He is there, but we are not happy with how he works and what opportunities he provides. Another is the optimization of the work of the memory subsystem of process groups (memcg).
    The way it works now does not suit us in terms of performance and resource allocation between containers. And one more optimization - for the FUSE file system data cache. The way it is implemented now does not allow achieving maximum performance on many types of load. And all this is done right away with an eye to integration into the Linux tree.
    There are also big plans for integrating OpenVZ with other projects, for example, it is almost all available as part of Fedora 19. People from OpenStack favorably look at the idea of ​​supporting OpenVZ in their services. Integration with CRIU - from the same series.
    There are some more interesting directions, which, by the way, are quite “drawn” to the title of subproject. But unfortunately I can’t tell about them yet.

    What fundamentally can be expected in the coming years in the field of virtualization?
    Before answering this question, I want to make a small clarification. Usually, the term “virtualization” refers to the technology of creating virtual machines, that is, hardware virtualization. Containers are, in a general sense, also virtualization, just at a different level, but as a rule they do not call it virtualization, and they say so - containers.
    So. It seems to me that in a few years virtual machines will differ in performance (for the worse, of course) from real ones so little that this minus will cease to outweigh all the advantages that virtualization provides. As a result, a virtual machine on any system will become not an option, but an integral component, and so naturally integrated into it that users will not even wonder if they work with a real or virtual system.
    The same thing will happen to containers, but on a different “front”. Now containers can be used as a shell for a separate process, application, and for the whole image of the operating system (without a kernel). In the future, the use of containers to create a virtual operating system should come to naught (virtual machines will fill this niche). And the use of containers to "isolate" services, both local and cloud, should become an invisibly standard.

    How did the development of the kernel developer take place? What difficulties arose at the initial stage?
    It all began, it seems, in the fourth year of the institute. I then worked on an academic distributed file system, and my research supervisor said that Parallels (then the company was also called SWsoft) gathered to make a commercial project out of this system, and sent to speak with his graduate student to join them. This postgraduate student worked in the Linux kernel team, they interviewed me and took me to the team with the words: “We will deal with this file system later, so far start from the kernel”. As a result, Parallels did not reach the system, so I remained a nuclear developer.
    The real difficulties arose when I had to write code not for the kernel that was developed inside Parallels, but for the core that Linus Torvalds does, and this was due to the fact that the code development models in Parallels and the community were radically different.
    At Parallels, this is more or less standard commercial code development. Community work is a completely different process. Its main feature, from my point of view, is that it is ... not quite a software development process. This is mainly communication of people who are interested in creating a large and complex program, and the actual development there is in second place. Awareness of this fact and adjustment for it were the main difficulties.

    What kernel subsystems are you working on?
    Almost all but drivers. In a sense, this is one of the problems - you can’t plunge deeply and thoroughly into any of the systems, you have to constantly monitor the development of all. True, there are “favorite” systems - for me it is a network code and a memory management subsystem.

    How is the process of including your patches in the kernel? What are the features of this process? What are the difficulties?
    The process looks the same for everyone. First you need to make the patch itself or a series of them, write comments on each (this, by the way, is a separate skill). Then the patch is sent to the mailing list, which discusses the desired kernel subsystem. A “copy” is made of a person who supports the subsystem (it is called maintainer in English), and people who understand it and can help evaluate the quality of work. Then you have to wait a bit. If everything is done well and everyone likes everything, the maintainer patch takes it to its repository. Then Linus includes the maintainer accumulated by all in his tree.
    This is the easiest way, but it can get complicated and drag out. If the change is large and complex, or the person who made it is inexperienced, then after sending the patch to the newsletter, discussion can begin. People who understand the appropriate code may find errors in the patch, they may say that the solution to the problem should be different, or they may ask you to make cosmetic changes, for example, rename a variable or write a more detailed comment. After that, the process should begin anew. The most important thing for the author of the patch here is not to accept criticism at your own expense, but to delve into its essence and, if you agree with everything, redo it.
    For example, when I started sending parts of container code to mailings, I had to rewrite the code from scratch several times. As a result, what Linus now has “about containers” does not look like a single line with what we have, although it does the same thing.

    Are there standards for code to be included in the Linux kernel? If so, could you give specific examples? Is deviation from standards allowed?
    There are certainly standards. Even written a special document - Kernel Coding Style. It contains all the requirements for registration. The requirements, it must be said, are very reasonable. It’s easy to get used to them, and it’s very difficult to start writing (and reading) differently later. Deviations from the standard, on the one hand, are allowed, but, on the other, they are all described in the standard itself, so we can say that it doesn’t.

    Do you have to write assembler for the Linux kernel?
    Sometimes. In general, the core is written as abstract as possible from architecture. It often happens that you write in SI, but in code that is architecturally specific.
    The result is new functionality that only works on one architecture. But the community has learned to cope with this, after a while a “connoisseur” of another platform appears who wants the new functionality to work for him, too, and completes it.

    How and when did you meet Linux? What distributions did you use?
    He met Linux in his second year at the institute. We had a whole semester there dedicated to Unix systems using Linux as an example. Somewhere a year after the first installation, Windows was finally demolished. By the way, when the time came to write a diploma, almost all classmates did it in Microsoft Word, and I in LaTeX. As a result, only my work looked worthy of protection from the point of view of design and layout.
    The first distribution I installed for myself was AspLinux, because then it was told to me by “more experienced colleagues” that the Russian language works normally in it and only in it. Then I tried Centos, Suse, now I use Fedora.

    What does your work day look like? How much time do you devote to programming, how much to design?
    Um ... I can’t even imagine what the average working day looks like. Typically, a work week is a salad of various “think,” “draw,” “hang around,” “talk,” “write to each other,” which I try to distribute in time so that there are as few switchings from one activity to another and people who expect me to take some action did not expect them to “idle”.

    What software do you use in your work? Work environment (GNOME, KDE, XFCE, etc.), email client (maybe a bunch of web interface / email client), web browser, editor, developer tools (e.g. git)?
    "Tuned" (as usual) Fedora Linux. Well, Parallels products, of course. KDE and Gnome I do not like, I immediately put myself FluxBox. Of the graphical applications I use Thunderbird, Chrome, Skype. Periodically, you need to draw slides for the next conference. So far, enough OpenOffice features.
    The rest (even cartoons for children) run in the terminal.
    VIM editor, as well as standard developer tools - make, git, svn, gcc, objdump, gdb.

    Do you use cloud services for work or personal purposes? If so, which ones?
    Besides google + and hangouts, almost nothing. I often look at all these services “from the other side” in order to get topics for reflection on our cloud products, or just for “general development”.

    What does the workplace look like? Are there any particular preferences in choosing the hardware?
    Table, chair, laptop. There is only one hardware preference - the IBM ThinkPad has a very comfortable keyboard and trackpoint.

    Do you use mobile devices for work purposes? If so, what do you prefer?
    The device (one) - yes, I bought myself a couple of years ago on one of HTC Sensation's business trips, so that I couldn’t get my laptop and a book reader with me into the load at airports all the time. Well, at the same time, he almost saved me a separate navigator. Since then, nothing has changed. In this regard, I am a classic “shoemaker without boots”. I don’t understand mobile gadgets as a consumer, home Linux is configured to a minimum and without frills, I don’t even connect the mobile Internet, because I can’t figure out why it can be useful to me “on the street”.

    Do you take part in specialized conferences? Do you make presentations?
    Yes, a couple of times a year I talk about something at Linux conferences. Sometimes I come without a report, but then the goal is to participate in a kind of “round table”. This is one of the aspects of community life, people communicate with each other most of the time via e-mail and periodically organize such “gatherings” to socialize and improve their understanding.

    Are you in the forums, in the chat rooms?
    But this is not. I have a rather specific range of programming interests, usually they don't ask about such things in forums.

    Often, developers say that they listen to one or another music while working. What do you think about it? What kind of music are you listening to?
    Depends on what I “work”. If this is something routine, such as viewing the newsletter per day, then you can do anything. If this is coding, about which it is already clear what exactly to code, then music without words, otherwise it is difficult to concentrate. For more intellectual activity, I prefer silence.
    And it seems to me that I don’t have any obvious preferences in music - these are classics, heavy music, and Soviet pop-rock set, and some random songs that I heard on the radio.

    What advice would you give to new kernel developers? How to reduce the "threshold" of entry into the development process?
    I heard a lot of advice on what to do and what not to do. It seems to me that they all boil down to the fact that the development of the Linux kernel is primarily communication in the "club of interests" and only in the second - the development itself. A kind of open social network system programming lovers.
    Therefore, the first thing to do is tune in to this style of communication. And everything else will follow.

    What books or sources can you recommend for those who start developing a kernel?
    I recommend just books, with very few exceptions, not to read about the kernel device - it changes too quickly, so any book becomes outdated in less than a year.

    It’s better to get comfortable with the principles of functioning of hardware platforms and operating systems and immediately “plunge” into reading code.
    But I can recommend the site kernelnewbies.org - for beginners in the kernel this will be at least an excellent and, most importantly, up-to-date collection of links to other sites, documentation and a certain amount of canonical literature on the kernel.

    Pavel Emelyanov was born in 1982. In 2004, he began working at SWsoft, which in 2007 was renamed Parallels. In 2005 he graduated from the Moscow Institute of Physics and Technology. In 2008 he defended his thesis "Mathematical modeling of memory consumption management processes in multi-user environments and virtualization systems", becoming a candidate of physical and mathematical sciences in the specialty "Mathematical modeling, numerical methods and program complexes".
    He is the architect of many Linux kernel-related systems at Parallels, including the OpenVZ and CRIU project leader. Organizes the interaction of the Parallels kernel development team with the Linux kernel community.

    Interviewed by Igor Shtompel, System Administrator magazine

    Also popular now: