“There is no boss here”: about working with Open Source and Apache Ignite in Sberbank Technologies
With the words “open source,” many people are either an enthusiast committing to a favorite project in the evenings, or a small company earning support from an open source product. But if you only think about them, then you will miss the important and interesting segment of the community. Once the words "enterprise" and "open source" seemed antonyms, and now large corporations are not only actively using OSS-projects, but also donating to them themselves.
Over time, Sbertech is increasingly manifesting himself in the OSS community, and we decided to ask them about it. How are the strict banking specifics combined with the open-air spirit of freedom? What are the requirements for open source, which may not be from other companies? Are there any employees in Sbertech who write open source code as their main work tasks? What are the plans and desires for the future? Anton Churaev , who oversees the direction of Free & Open Source, told us about all this and not only.
Oleg: Hi, Anton. Let's introduce a little habrovchan. Tell about yourself: who are you, what are you doing?
Anton: I am an engineer who, however, develops only in his spare time. Now I am building in Sbertech practices and competencies for the development and application of Free & Open Source products. You need to understand that these are slightly different things.
- Yes, I understand, a couple of times when communicating with Stallman, I called Free as Open and after that I listened to such a lecture that I remembered for the rest of my life :-) Well, what is your position?
- Curator of Free & Open Source Development. And open-sourced enthusiast :) - Can you tell a
little more about “competencies in the implementation of Open Source”? Sounds like some kind of secret knowledge.
- Few people imagine what open source means for corporations. On the one hand, this is innovation and the absence of the need to develop a commodity, a concentration on the competitive advantages of its products, reuse and cost reduction. Often these are projects that have already become industry standards. Take the same Hadoop - he is widely known, everyone knows him, he has long been a standard. Or the most common databases - in the top five are three open source products - MySQL, PostgreSQL and MongoDB.
But few people think that using OpenSource implies a lot of hidden costs. This is not to say that we have found something with open source and solved all our problems. For example, there are big questions on "legal hygiene". When working with vendor software, everything is very clear: you took a license, work on it and have support. When working with Open Source, a lot of things are left to developers and users. In this case, legal and legal issues arise in one of the first places. In addition, in Russia there are nuances. If all over the world the concept of intellectual property is quite strongly developed and everyone understands that it is very important that there is a specific owner, then historically in Russia everything turned out differently. Here we treat intellectual property not so carefully and respectfully, although it is extremely wrong.
- Can you clarify? What is the legal status of GPL type licenses in Russia? For example, the GPL does not allow modifications under local law and does not specify territorial restrictions. Therefore, such a contract is not compatible with the legal regime established in the territory of the Russian Federation, and this is very, very bad.
- I do not want to share licenses for some zones. Sberbank is a global company, so software can be used both in the United States and in the European Union. And, as far as I understand (I am not a lawyer, but to the best of my knowledge), in case of violation of the limitations of the licensor’s rights in a territory, for example, the United States, we will respond under US law. Given this, you need to be very careful about securing rights and fulfilling the requirements for someone else's intellectual property. To respect the authors who allowed us to use their work, due to which we accelerated, optimized solutions, solved our tasks and eventually provided high-quality service to our clients. Let's respect the rights and requirements. This is the first task.
- And the second?
- The second task is information security. It is clear that the majority of licenses contain a disclaimer stating that the author / developer / contributor is not responsible for the possible harm that will be caused during the operation of this software. This is right, it is a responsibility that passes to the consumer and requires him to mature. Everything is free.
You have to pay for this responsibility and, of course, work with these risks. Not all companies can do this. We also have a very strong department of information security - we were lucky. Therefore, we take seriously the presence of vulnerabilities and malicious code in general. Anyone who plans to use Open Source must take into account all the risks - not only legal, but also in terms of information security.
- What kind of licenses do you like?
- Academic.
- O! Come more specifically. Here there is MIT / BSD, etc., and there are virus copyleft licenses such as Affero GPL. What of it?
- Oh well. You can not love or dislike the license. The product is made for a specific task and will be used in a certain way. When using open source, your task is to ensure that you provide your product or service without violating the rights of third parties. At the same time, of course, you can use copylefts (for example, the GPL), if you ensure their use in such a way that they do not violate restrictions and anyone's rights. Of course, there are less difficulties when using academic licenses, simply because they carry fewer restrictions and, therefore, they are easier to follow. For academic purposes, I call “academic” MIT, BSD, Apache, and others.
- Okay, and information security has to deal with ordinary developers? Or is it allocated to a separate department?
- All developers should understand the basics of information security and the principles of its provision for their systems. But in our case, we work with individual developers who specialize in information security threats. And we turn to them not only on the analysis of open-source products, but also on the analysis of algorithms, design solutions.
- It is clear that these special security people know everything. And what does an ordinary developer need to know in this regard? What are the basic points?
- Model of the emergence of threats, channel protection, data protection. What is susceptible to threats: maybe this is a user interface or data transmission over the network (everything is distributed, so this is a very important question). Basic tools like encryption, SSL, TLS, authentication, authorization, work with tokens and so on. Many do not need to know.
- Rumor has it that you have something to do with Apache Ignite :-)
- In terms of content-binding, this is the main project I am currently involved in. Participation in Apache Ignite belongs to my second task - to ensure a balanced investment in Free and Open Source projects. This implies both the proper use of products (it is clear that the use of libraries is an investment, we, as users, increase the attractiveness of the product), as well as the development and supplying.
For me, probably, this task is even more significant. We pay tribute to the products we use in our company and thanks to which we have built a lot of products and systems. We are trying to improve them and ensure the possibility of using them in companies like ours: optimize, bring to the enterprise state.
Apache Ignite is not the only project, but we are very intensively contributing to it, because one of the key platforms in the bank is built on the basis of Apache Ignite. Ignite is a distributed computing grid that allows you to store and process data in memory, and in fact is the basis of our business IT landscape. Therefore, we are extremely interested in its development.
“Many people know that GridGain is used in Sbertech, and you are talking about Apache Ignite.” What is the difference between them?
- GridGain is an open core-product built on the basis of an open core, which is Ignite. And GridGain is a set of plug-ins to this core, which simplify maintenance and operation procedures, provide a number of important information security and reliability requirements. But, in essence, the kernel is the most significant part, and the plugins allow you to exploit all this in a real enterprise. And the bank has already operated GridGain.
- Since Ignite is open, you can talk about it a bit, right? Do you only exploit it or do you finish something, interact with developers?
- We are intensively finalizing it. The directions of the tasks are clearly defined, for example, ensuring performance with the specifics of Sberbank: large scale, ocean data, high operational activity. Therefore, it should be fast and plenty. By this I mean both latency and throughput.
The second is to ensure reliability, i.e. availability and fault tolerance.
The third is operational efficiency, TCO management. Given the scale of the Savings Bank, even a small reduction in the use of resources, for example, drives by a certain percentage, on our scales gives tremendous savings.
And the fourth - functional development tasks. In fact, the main thing is the development of interfaces and integration with other components of the Sberbank technological stack. This is useful and important from the point of view of building a mature and integrated IT architecture.
Separately, the task is to eliminate debt and defects (which are always). It probably can be attributed to reliability.
- Let's go over these points for clarification. You say that you are working on improving performance, latency, throughput, that's all. The question is - does Ignite have any problems with this? In a sense, is there something to refine or is it already an ideal product?
- No, you can not consider the product perfect. In each release, we drive both general benchmarks and microbench marks on specific components, all the time we are working on performance - in this matter we must not stop. The task is difficult, because the components and solutions are already quite dimmed, the performance is almost at the limit of iron. This adds complexity, but there is always room for improvement. We have different use cases, new user tasks appear, in which there is the possibility of optimization. For example, optimize the streamer for the specific nature of the data. There are tasks to optimize the network layer, which, again, depends on individual cases. Therefore, you should always keep your finger on the pulse.
- You said you contribute back to the community. And all these decisions about various cases and optimization for them - some kind of tradeoff. When we take our tradeoffs and bring it into the community, it may turn out that people in the community have slightly different conditions and different priorities. How to organize the interaction and still legalize the code that is needed for your cases?
- Other customers with other tasks. Absolutely true, this is a standard problem. Everything depends heavily on the solution architecture. If the solution contains, for example, the ability to make plug-ins, plug-ins, libraries for different user solutions, it will be possible to get out. For example, if there is a comparator, the user can always develop a solution that will pass this comparator to the input, and this will solve the problem based on specific conditions. Again, the possibilities are very dependent on the architecture. It is wrong to just roughly draw the code and sculpt for our task without thinking about other clients - such pull requests do not pass review.
Everyone understands what an open source project is, and in general, you can influence it. Of course, there are communities in which there are clearly corporations that influence development in their own interests, but if we play pure open source, then it will be correctly compared with meritocracy (the power of worthy). Prove that your decision is good, and then it will be made. Acting, as often happens in a closed development, that is, from the position “I am the boss, I said so,” will not work.
- One of the most interesting cases that Sbertech told in public - the Unified Semantic Layer. A huge amount of data spread on the in-memory grid. How much has this affected the open part of Ignite and how interesting are these developments for the community?
- Yes, such developments are underway, and we are very intensively working on tasks to ensure scaling and accessibility. We found cases in which the current topology control scheme is not optimal, because its time complexity grows from the number of nodes not quite as we would like. This somewhat complicates the achievement of the goal.
- As far as I remember, the cluster architecture is a ring. That is, when we join the ring, then at the beginning we go to the coordinator and then we go around the ring until we find the tail. And the more elements, the more time, right?
- Yes, sort of. At the same time, as the number of nodes in the topology increases, both the size of the messages that are transmitted along the ring and the number of transitions between the participants grow. This is not to say that the ring is a bad decision, but in some cases it is not suitable. Therefore, starting from the end of 2017, we in the community are finalizing the topology management, so that users can choose a topology management scheme: a ring (sometimes it fits perfectly) or a star on the Zookeeper.
- And where did the ring come from? Why is it? Where is it perfect?
- This is a great solution on topologies of 100-200 nodes in one data center. Allows you to simply and reliably synchronize all nodes, they just go in order. If we turn to a star, they start to work in parallel, faster, but at the same time it becomes much more difficult to synchronize them. That is, the ring can be more stable and reliable, agree?
- Yes of course. And you can make it so that this topology can be changed by some parameter in the config, how is the setting?
- Yes, we are doing this now, including both topologies in the release. Probably, the proposed implementations do not cover all cases of users, and as soon as new ones appear, we will try to ensure their efficient processing.
- As far as I understand, this is a rather complicated refinement. And this refinement is done by people in Sbertech, during working hours, or in the evenings for your pleasure?
- This makes the community, which includes the staff of the SBT, whose main task during working hours is to contribute to this project. The topology challenge involves one of the key decisions in the core of the product, so the main burden was placed on DiscoverySPI's maintainers, but I hope that the participation of our developers also had a positive effect on the result.
- Well, that is, people who solve the problem during working hours, but at the same time are members of the community.
- Yes, the most significant part of the work of our developers is in the community. But I see from our guys and such commits that appeared in an hour or two or three nights.
- And these employees will have no problem from the fact that they work in a bank, on a closed system, and at the same time commit to open-source?
- No, it will not. All participants are official corporate contributors. The creation of a direction and a decision on investments was made at the level of the company's management, and yes - this group of dedicated corporate contributors who, in accordance with all the norms of the company and the TC, carry out the development of Open Source products in the company's interests. Yes - this is the development and Open Source, yes - this is during working hours, and during non-working hours too, but this is already if the community strongly requests.
- We are now talking about some external affairs, which solves the community. But most likely, the company needs to do its own integration, improve it for some of its cases ... Have you written a lot of your own? Or is it just a little dopilki?
- If we talk about the Apache Ignite project, in the last quarter our contribution to the project amounted to 8-10 percent of all changes, and we are striving to increase this percentage. We wrote a lot, and this is not only the development and optimization of the existing functionality, we are also working on a new functionality for the project. For the community, this is a challenge, and for us, responsibility, since after its inclusion, the community in some way has the task of supporting it.
Tasks can appear not only from the community, but also from users within the company: architects, development and maintenance teams. The development of the project for these tasks also significantly affects the product.
- But, let's say, there were several reports from the Sbertekhovskaya program of the PRBR regarding their “special Feng Shui”. Do I need to write any additional tula and admin to support this?
- Interfaces for operation are constantly evolving. The management console of the same Oracle is more familiar to maintain, and so far has more functionality. Whether it is necessary to reproduce it completely is another question.
- And in the open form, you can see the management console?
- Yes of course. Web Console is published, Visor, CLI - everything is public.
- And if you look at it more globally, what directions and goals are there at all?
- Now we are more focused on the development of Apache Ignite, which meets the priorities of the company. But our technological stack doesn't end there. We work with a lot of open source projects where we see opportunities for development, and we have something to offer to improve these projects. I hope that they will be interesting not only to us, but also to other users. For each project, we determine the required amount of our participation (from defect correction to architecture change), we assess whether we have the necessary competencies, readiness and interest of the community in cooperation. As a result of all this, we understand the amount of resources required from our side. For the project, the value of our participation in the fact that we can offer cases of a very large bank. This project can give a serious impetus to growth. We already have such cases.
- You said that you can bring yuzkeys large bank. How will these user cases differ from something else?
- The main difference of Sberbank from the rest is the requirements for reliability, that is, availability, durability, fault tolerance, etc.
- And security, I guess.
- Yes, security is a separate issue. We hope that we can bring the Open Source project and go beyond the adaptation to these requirements. Otherwise, he will search for user cases for a very long time. Not everyone is faced with the requirements that are in the bank.
- Are there any popular products that everyone knows about, which still have to be cut and sawed to such a scale? For example, any distributed file storage, ceph?
- Well, yes, ceph is a good example. The project is good, wonderful, very mature, but it can still be improved.
- Do you use any virtuals for developers inside? How is this controlled?
- OpenStack.
- As far as I understand, OpenStack is such a thing that you can actually modify. Do you do something with it, or how you put it in vanilla, does it work?
“We have not worked on OpenStack yet, but definitely this is an interesting direction, like Cloud Foundry, containers.
“What about containers?”
- We have great with them :-) We understand that on our scale, to ensure effective utilization and resource management, we must implement (and implement now) the containerization of applications. It also raises the question of involvement in the development of these projects, because enhancing competitive advantages is a task that is useful both to us and to the project itself.
- Let's talk about people.
“Given the ambitions and current expectations of the bank, at some point we will turn into one of the major corporate contributors in Open Source and will be visible at the global level.
But there is a problem that there are few strong system developers on the market. And in the company too, as the bank has always focused more on application development, business application development.
- That is, ready-made technologies in a special way were glued together and specific business problems were solved?
- Yes, rather business problems were solved: issuing a loan, calculating scoring, etc. Of course, we have made and are making unique decisions in terms of customer experience, performance, reliability, security. But we are approaching the limit of solutions offered by vendors and can face the approach of competitors. Therefore, now we are talking about developing your own platform, intensive search and use of innovations of Open Source projects, which, in turn, requires examination in system development.
Our country has a good base in terms of student training. They are prepared for what is needed in system development. They are not prepared to write business processes, they are prepared to look for and evaluate solutions, they give excellent mathematical and algorithmic training. This knowledge and skills are extremely important to us now. In the tasks we are solving now, much that would be of secondary importance in applied development comes to the fore, mainly questions of the effectiveness of solutions.
But, unfortunately, since system development has not been in demand for a long time (with the exception of large IT companies, such as Kaspersky, Mail, Yandex) - we will be growing for a long time at least hundreds of contributors. There are very few developers who can effectively engage in system development, although with a good academic base, the necessary skills can be increased.
The second is the experience and deep knowledge of programming languages.
The third is the communication requirements, because we work in the community. There is no boss here who says “do” to the developer, there is a community here that says “maybe, I will accept” or “rather, I will not accept”. The problem is different, and we restructured for it. From the developer, no one requires the code, he must convince the community to accept his work, his contribution. It is much harder to take criticism, communicate, review, explain and be public. Being useful to other members of the community is a basic requirement that we make.
- And what kind of motivation should a person have to do with all this? Many come to work, work from eight to five with a mandatory lunch on the instructions of the head ... and everything that you described is not very good in this scheme.
- Motivation in development. We are closely looking at candidates, trying to determine whether they understand themselves, and how well they understand where they want to go and why we need them. We give developers the opportunity to develop, to have a public reputation at the global level, which is not zeroed when switching to another employer (and this reputation is much more valuable). For such developers, values are naturally aligned with our team and company, they understand why they want to work with us, and not just spend working time in the office.
But those who can work effectively in an open community are very, very few.
Community work and the development of Open Source are different from the work of other developers of the company. At our contributors, the processes are more integrated with the community, we primarily look at it. And only then we are trying to restructure our internal corporate processes to the expectations of the community. If others solve problems of internal bureaucracy, then we solve problems of interaction with the community.
- Why are you doing this? What is your motivation?
- For me, this is a challenge and an opportunity to positively influence Sberbank's IT, increase its competitiveness. But, which is also extremely important, this is an opportunity to influence in general the development of IT by contributing to open projects. Of course, this is all related to the company's main business goals - brand development, increasing ROI, building a Platform, etc., therefore this trend is in demand by the bank's management.
- In general, system programming is somehow not very popular in Russia. I often hear what people say: if I do system programming, then as a result I will not find a job anywhere. It is useful for everyone to create websites, to know some basic things like Spring and Hibernate, and if I come to study multithreading, then oh, no one in this country needs it, but I still have to go abroad. And such an activity like yours can, at a minimum, increase the popularity of these competencies. If only because with them you can go to you.
“Perhaps that is why there are more good applied developers on the market than system developers.” I am very glad that we managed to gather the current team, where all the guys are very bright. Striking how they think and work, they can solve almost any task. I hope that we can attract even more talented developers. Therefore, we cannot say for sure that nobody in Russia needs system developers.
Over time, Sbertech is increasingly manifesting himself in the OSS community, and we decided to ask them about it. How are the strict banking specifics combined with the open-air spirit of freedom? What are the requirements for open source, which may not be from other companies? Are there any employees in Sbertech who write open source code as their main work tasks? What are the plans and desires for the future? Anton Churaev , who oversees the direction of Free & Open Source, told us about all this and not only.
Oleg: Hi, Anton. Let's introduce a little habrovchan. Tell about yourself: who are you, what are you doing?
Anton: I am an engineer who, however, develops only in his spare time. Now I am building in Sbertech practices and competencies for the development and application of Free & Open Source products. You need to understand that these are slightly different things.
- Yes, I understand, a couple of times when communicating with Stallman, I called Free as Open and after that I listened to such a lecture that I remembered for the rest of my life :-) Well, what is your position?
- Curator of Free & Open Source Development. And open-sourced enthusiast :) - Can you tell a
little more about “competencies in the implementation of Open Source”? Sounds like some kind of secret knowledge.
- Few people imagine what open source means for corporations. On the one hand, this is innovation and the absence of the need to develop a commodity, a concentration on the competitive advantages of its products, reuse and cost reduction. Often these are projects that have already become industry standards. Take the same Hadoop - he is widely known, everyone knows him, he has long been a standard. Or the most common databases - in the top five are three open source products - MySQL, PostgreSQL and MongoDB.
But few people think that using OpenSource implies a lot of hidden costs. This is not to say that we have found something with open source and solved all our problems. For example, there are big questions on "legal hygiene". When working with vendor software, everything is very clear: you took a license, work on it and have support. When working with Open Source, a lot of things are left to developers and users. In this case, legal and legal issues arise in one of the first places. In addition, in Russia there are nuances. If all over the world the concept of intellectual property is quite strongly developed and everyone understands that it is very important that there is a specific owner, then historically in Russia everything turned out differently. Here we treat intellectual property not so carefully and respectfully, although it is extremely wrong.
- Can you clarify? What is the legal status of GPL type licenses in Russia? For example, the GPL does not allow modifications under local law and does not specify territorial restrictions. Therefore, such a contract is not compatible with the legal regime established in the territory of the Russian Federation, and this is very, very bad.
- I do not want to share licenses for some zones. Sberbank is a global company, so software can be used both in the United States and in the European Union. And, as far as I understand (I am not a lawyer, but to the best of my knowledge), in case of violation of the limitations of the licensor’s rights in a territory, for example, the United States, we will respond under US law. Given this, you need to be very careful about securing rights and fulfilling the requirements for someone else's intellectual property. To respect the authors who allowed us to use their work, due to which we accelerated, optimized solutions, solved our tasks and eventually provided high-quality service to our clients. Let's respect the rights and requirements. This is the first task.
- And the second?
- The second task is information security. It is clear that the majority of licenses contain a disclaimer stating that the author / developer / contributor is not responsible for the possible harm that will be caused during the operation of this software. This is right, it is a responsibility that passes to the consumer and requires him to mature. Everything is free.
You have to pay for this responsibility and, of course, work with these risks. Not all companies can do this. We also have a very strong department of information security - we were lucky. Therefore, we take seriously the presence of vulnerabilities and malicious code in general. Anyone who plans to use Open Source must take into account all the risks - not only legal, but also in terms of information security.
- What kind of licenses do you like?
- Academic.
- O! Come more specifically. Here there is MIT / BSD, etc., and there are virus copyleft licenses such as Affero GPL. What of it?
- Oh well. You can not love or dislike the license. The product is made for a specific task and will be used in a certain way. When using open source, your task is to ensure that you provide your product or service without violating the rights of third parties. At the same time, of course, you can use copylefts (for example, the GPL), if you ensure their use in such a way that they do not violate restrictions and anyone's rights. Of course, there are less difficulties when using academic licenses, simply because they carry fewer restrictions and, therefore, they are easier to follow. For academic purposes, I call “academic” MIT, BSD, Apache, and others.
- Okay, and information security has to deal with ordinary developers? Or is it allocated to a separate department?
- All developers should understand the basics of information security and the principles of its provision for their systems. But in our case, we work with individual developers who specialize in information security threats. And we turn to them not only on the analysis of open-source products, but also on the analysis of algorithms, design solutions.
- It is clear that these special security people know everything. And what does an ordinary developer need to know in this regard? What are the basic points?
- Model of the emergence of threats, channel protection, data protection. What is susceptible to threats: maybe this is a user interface or data transmission over the network (everything is distributed, so this is a very important question). Basic tools like encryption, SSL, TLS, authentication, authorization, work with tokens and so on. Many do not need to know.
- Rumor has it that you have something to do with Apache Ignite :-)
- In terms of content-binding, this is the main project I am currently involved in. Participation in Apache Ignite belongs to my second task - to ensure a balanced investment in Free and Open Source projects. This implies both the proper use of products (it is clear that the use of libraries is an investment, we, as users, increase the attractiveness of the product), as well as the development and supplying.
For me, probably, this task is even more significant. We pay tribute to the products we use in our company and thanks to which we have built a lot of products and systems. We are trying to improve them and ensure the possibility of using them in companies like ours: optimize, bring to the enterprise state.
Apache Ignite is not the only project, but we are very intensively contributing to it, because one of the key platforms in the bank is built on the basis of Apache Ignite. Ignite is a distributed computing grid that allows you to store and process data in memory, and in fact is the basis of our business IT landscape. Therefore, we are extremely interested in its development.
“Many people know that GridGain is used in Sbertech, and you are talking about Apache Ignite.” What is the difference between them?
- GridGain is an open core-product built on the basis of an open core, which is Ignite. And GridGain is a set of plug-ins to this core, which simplify maintenance and operation procedures, provide a number of important information security and reliability requirements. But, in essence, the kernel is the most significant part, and the plugins allow you to exploit all this in a real enterprise. And the bank has already operated GridGain.
- Since Ignite is open, you can talk about it a bit, right? Do you only exploit it or do you finish something, interact with developers?
- We are intensively finalizing it. The directions of the tasks are clearly defined, for example, ensuring performance with the specifics of Sberbank: large scale, ocean data, high operational activity. Therefore, it should be fast and plenty. By this I mean both latency and throughput.
The second is to ensure reliability, i.e. availability and fault tolerance.
The third is operational efficiency, TCO management. Given the scale of the Savings Bank, even a small reduction in the use of resources, for example, drives by a certain percentage, on our scales gives tremendous savings.
And the fourth - functional development tasks. In fact, the main thing is the development of interfaces and integration with other components of the Sberbank technological stack. This is useful and important from the point of view of building a mature and integrated IT architecture.
Separately, the task is to eliminate debt and defects (which are always). It probably can be attributed to reliability.
- Let's go over these points for clarification. You say that you are working on improving performance, latency, throughput, that's all. The question is - does Ignite have any problems with this? In a sense, is there something to refine or is it already an ideal product?
- No, you can not consider the product perfect. In each release, we drive both general benchmarks and microbench marks on specific components, all the time we are working on performance - in this matter we must not stop. The task is difficult, because the components and solutions are already quite dimmed, the performance is almost at the limit of iron. This adds complexity, but there is always room for improvement. We have different use cases, new user tasks appear, in which there is the possibility of optimization. For example, optimize the streamer for the specific nature of the data. There are tasks to optimize the network layer, which, again, depends on individual cases. Therefore, you should always keep your finger on the pulse.
- You said you contribute back to the community. And all these decisions about various cases and optimization for them - some kind of tradeoff. When we take our tradeoffs and bring it into the community, it may turn out that people in the community have slightly different conditions and different priorities. How to organize the interaction and still legalize the code that is needed for your cases?
- Other customers with other tasks. Absolutely true, this is a standard problem. Everything depends heavily on the solution architecture. If the solution contains, for example, the ability to make plug-ins, plug-ins, libraries for different user solutions, it will be possible to get out. For example, if there is a comparator, the user can always develop a solution that will pass this comparator to the input, and this will solve the problem based on specific conditions. Again, the possibilities are very dependent on the architecture. It is wrong to just roughly draw the code and sculpt for our task without thinking about other clients - such pull requests do not pass review.
Everyone understands what an open source project is, and in general, you can influence it. Of course, there are communities in which there are clearly corporations that influence development in their own interests, but if we play pure open source, then it will be correctly compared with meritocracy (the power of worthy). Prove that your decision is good, and then it will be made. Acting, as often happens in a closed development, that is, from the position “I am the boss, I said so,” will not work.
- One of the most interesting cases that Sbertech told in public - the Unified Semantic Layer. A huge amount of data spread on the in-memory grid. How much has this affected the open part of Ignite and how interesting are these developments for the community?
- Yes, such developments are underway, and we are very intensively working on tasks to ensure scaling and accessibility. We found cases in which the current topology control scheme is not optimal, because its time complexity grows from the number of nodes not quite as we would like. This somewhat complicates the achievement of the goal.
- As far as I remember, the cluster architecture is a ring. That is, when we join the ring, then at the beginning we go to the coordinator and then we go around the ring until we find the tail. And the more elements, the more time, right?
- Yes, sort of. At the same time, as the number of nodes in the topology increases, both the size of the messages that are transmitted along the ring and the number of transitions between the participants grow. This is not to say that the ring is a bad decision, but in some cases it is not suitable. Therefore, starting from the end of 2017, we in the community are finalizing the topology management, so that users can choose a topology management scheme: a ring (sometimes it fits perfectly) or a star on the Zookeeper.
- And where did the ring come from? Why is it? Where is it perfect?
- This is a great solution on topologies of 100-200 nodes in one data center. Allows you to simply and reliably synchronize all nodes, they just go in order. If we turn to a star, they start to work in parallel, faster, but at the same time it becomes much more difficult to synchronize them. That is, the ring can be more stable and reliable, agree?
- Yes of course. And you can make it so that this topology can be changed by some parameter in the config, how is the setting?
- Yes, we are doing this now, including both topologies in the release. Probably, the proposed implementations do not cover all cases of users, and as soon as new ones appear, we will try to ensure their efficient processing.
- As far as I understand, this is a rather complicated refinement. And this refinement is done by people in Sbertech, during working hours, or in the evenings for your pleasure?
- This makes the community, which includes the staff of the SBT, whose main task during working hours is to contribute to this project. The topology challenge involves one of the key decisions in the core of the product, so the main burden was placed on DiscoverySPI's maintainers, but I hope that the participation of our developers also had a positive effect on the result.
- Well, that is, people who solve the problem during working hours, but at the same time are members of the community.
- Yes, the most significant part of the work of our developers is in the community. But I see from our guys and such commits that appeared in an hour or two or three nights.
- And these employees will have no problem from the fact that they work in a bank, on a closed system, and at the same time commit to open-source?
- No, it will not. All participants are official corporate contributors. The creation of a direction and a decision on investments was made at the level of the company's management, and yes - this group of dedicated corporate contributors who, in accordance with all the norms of the company and the TC, carry out the development of Open Source products in the company's interests. Yes - this is the development and Open Source, yes - this is during working hours, and during non-working hours too, but this is already if the community strongly requests.
- We are now talking about some external affairs, which solves the community. But most likely, the company needs to do its own integration, improve it for some of its cases ... Have you written a lot of your own? Or is it just a little dopilki?
- If we talk about the Apache Ignite project, in the last quarter our contribution to the project amounted to 8-10 percent of all changes, and we are striving to increase this percentage. We wrote a lot, and this is not only the development and optimization of the existing functionality, we are also working on a new functionality for the project. For the community, this is a challenge, and for us, responsibility, since after its inclusion, the community in some way has the task of supporting it.
Tasks can appear not only from the community, but also from users within the company: architects, development and maintenance teams. The development of the project for these tasks also significantly affects the product.
- But, let's say, there were several reports from the Sbertekhovskaya program of the PRBR regarding their “special Feng Shui”. Do I need to write any additional tula and admin to support this?
- Interfaces for operation are constantly evolving. The management console of the same Oracle is more familiar to maintain, and so far has more functionality. Whether it is necessary to reproduce it completely is another question.
- And in the open form, you can see the management console?
- Yes of course. Web Console is published, Visor, CLI - everything is public.
- And if you look at it more globally, what directions and goals are there at all?
- Now we are more focused on the development of Apache Ignite, which meets the priorities of the company. But our technological stack doesn't end there. We work with a lot of open source projects where we see opportunities for development, and we have something to offer to improve these projects. I hope that they will be interesting not only to us, but also to other users. For each project, we determine the required amount of our participation (from defect correction to architecture change), we assess whether we have the necessary competencies, readiness and interest of the community in cooperation. As a result of all this, we understand the amount of resources required from our side. For the project, the value of our participation in the fact that we can offer cases of a very large bank. This project can give a serious impetus to growth. We already have such cases.
- You said that you can bring yuzkeys large bank. How will these user cases differ from something else?
- The main difference of Sberbank from the rest is the requirements for reliability, that is, availability, durability, fault tolerance, etc.
- And security, I guess.
- Yes, security is a separate issue. We hope that we can bring the Open Source project and go beyond the adaptation to these requirements. Otherwise, he will search for user cases for a very long time. Not everyone is faced with the requirements that are in the bank.
- Are there any popular products that everyone knows about, which still have to be cut and sawed to such a scale? For example, any distributed file storage, ceph?
- Well, yes, ceph is a good example. The project is good, wonderful, very mature, but it can still be improved.
- Do you use any virtuals for developers inside? How is this controlled?
- OpenStack.
- As far as I understand, OpenStack is such a thing that you can actually modify. Do you do something with it, or how you put it in vanilla, does it work?
“We have not worked on OpenStack yet, but definitely this is an interesting direction, like Cloud Foundry, containers.
“What about containers?”
- We have great with them :-) We understand that on our scale, to ensure effective utilization and resource management, we must implement (and implement now) the containerization of applications. It also raises the question of involvement in the development of these projects, because enhancing competitive advantages is a task that is useful both to us and to the project itself.
- Let's talk about people.
“Given the ambitions and current expectations of the bank, at some point we will turn into one of the major corporate contributors in Open Source and will be visible at the global level.
But there is a problem that there are few strong system developers on the market. And in the company too, as the bank has always focused more on application development, business application development.
- That is, ready-made technologies in a special way were glued together and specific business problems were solved?
- Yes, rather business problems were solved: issuing a loan, calculating scoring, etc. Of course, we have made and are making unique decisions in terms of customer experience, performance, reliability, security. But we are approaching the limit of solutions offered by vendors and can face the approach of competitors. Therefore, now we are talking about developing your own platform, intensive search and use of innovations of Open Source projects, which, in turn, requires examination in system development.
Our country has a good base in terms of student training. They are prepared for what is needed in system development. They are not prepared to write business processes, they are prepared to look for and evaluate solutions, they give excellent mathematical and algorithmic training. This knowledge and skills are extremely important to us now. In the tasks we are solving now, much that would be of secondary importance in applied development comes to the fore, mainly questions of the effectiveness of solutions.
But, unfortunately, since system development has not been in demand for a long time (with the exception of large IT companies, such as Kaspersky, Mail, Yandex) - we will be growing for a long time at least hundreds of contributors. There are very few developers who can effectively engage in system development, although with a good academic base, the necessary skills can be increased.
The second is the experience and deep knowledge of programming languages.
The third is the communication requirements, because we work in the community. There is no boss here who says “do” to the developer, there is a community here that says “maybe, I will accept” or “rather, I will not accept”. The problem is different, and we restructured for it. From the developer, no one requires the code, he must convince the community to accept his work, his contribution. It is much harder to take criticism, communicate, review, explain and be public. Being useful to other members of the community is a basic requirement that we make.
- And what kind of motivation should a person have to do with all this? Many come to work, work from eight to five with a mandatory lunch on the instructions of the head ... and everything that you described is not very good in this scheme.
- Motivation in development. We are closely looking at candidates, trying to determine whether they understand themselves, and how well they understand where they want to go and why we need them. We give developers the opportunity to develop, to have a public reputation at the global level, which is not zeroed when switching to another employer (and this reputation is much more valuable). For such developers, values are naturally aligned with our team and company, they understand why they want to work with us, and not just spend working time in the office.
But those who can work effectively in an open community are very, very few.
Community work and the development of Open Source are different from the work of other developers of the company. At our contributors, the processes are more integrated with the community, we primarily look at it. And only then we are trying to restructure our internal corporate processes to the expectations of the community. If others solve problems of internal bureaucracy, then we solve problems of interaction with the community.
- Why are you doing this? What is your motivation?
- For me, this is a challenge and an opportunity to positively influence Sberbank's IT, increase its competitiveness. But, which is also extremely important, this is an opportunity to influence in general the development of IT by contributing to open projects. Of course, this is all related to the company's main business goals - brand development, increasing ROI, building a Platform, etc., therefore this trend is in demand by the bank's management.
- In general, system programming is somehow not very popular in Russia. I often hear what people say: if I do system programming, then as a result I will not find a job anywhere. It is useful for everyone to create websites, to know some basic things like Spring and Hibernate, and if I come to study multithreading, then oh, no one in this country needs it, but I still have to go abroad. And such an activity like yours can, at a minimum, increase the popularity of these competencies. If only because with them you can go to you.
“Perhaps that is why there are more good applied developers on the market than system developers.” I am very glad that we managed to gather the current team, where all the guys are very bright. Striking how they think and work, they can solve almost any task. I hope that we can attract even more talented developers. Therefore, we cannot say for sure that nobody in Russia needs system developers.