July 31, 2018 at 13:21“There is no boss here”: about working with Open Source and Apache Ignite in Sberbank Technologies
At the words "open source", many seem to be either an enthusiast who commits in the evenings to their favorite project, or a small company that makes money by supporting an open product. But if you think only about them, then you will miss an important and interesting segment of the community. Once the words "enterprise" and "open source" seemed to be antonyms, but now large corporations not only actively use OSS-projects, but they themselves will contribute to them.
Over time, Sbertech has been increasingly active in the OSS community, and we decided to ask them about it. How does strict banking specificity combine with the open source spirit of freedom? What are the requirements for Open Source that other companies might not have? Are there any employees in Sbertech who write open source as their main work tasks? What are your plans and desires for the future? Anton Churaev , who oversees Free & Open Source, told us about all this and more.
Oleg: Hi Anton. Let's introduce you to the Khabrovites a little. Tell about yourself: who are you, what are you doing?
Anton: I am an engineer who, however, develops only in his spare time. Now I am building in Sbertech practices and competencies for the development and application of Free & Open Source products. You need to understand that these are slightly different things.
- Yes, I understand, a couple of times when talking with Stallman I called Free as Open and after that I listened to such a lecture that I remembered for a lifetime :-) Well, what is your position?
- Curator of the development of Free & Open Source. And open source enthusiast :)
- Can you decode a little more about the "competencies for the implementation of Open Source"? It sounds like some kind of secret knowledge.
- Few people imagine what Open Source means for corporations. On the one hand, these are innovations and the absence of the need to develop commodity, focus on the competitive advantages of their products, reuse and reduce costs. Often these are projects that have actually become industry standards. Take the same Hadoop - everyone knows it, everyone knows it, it has long been a standard. Or the most common databases - the top five are three open source products - MySQL, PostgreSQL and MongoDB.
But few people think that using OpenSource involves a lot of hidden costs. This is not to say that we found something open source and solved all our problems. For example, there are big questions about “legal hygiene”. When working with vendor software, everything is very clear: I took a license, you work on it and use support. When working with Open Source, much is left at the mercy of developers and users. In this case, legal and legal issues are one of the first places. In addition, in Russia there are nuances. If all over the world the concept of intellectual property is quite developed and everyone understands that it is very important that there is a specific owner, then historically in Russia everything turned out differently. Here we are not so careful and respectful of intellectual property, although this is extremely wrong.
- Can I clarify? What is the legal status of GPL licenses in Russia? For example, the GPL does not allow modifications to local laws and does not indicate territorial restrictions. Therefore, such an agreement is not compatible with the legal regime established in the territory of the Russian Federation, and this is very, very bad.
- I do not want to divide licenses into some zones. Sberbank is a global company, so the software can be used both in the United States and in the European Union. And, as I understand it (I am not a lawyer, but to the best of my knowledge), in case of violation of the restrictions of the licenses of the copyright holder in the territory of, for example, the USA, we will be responsible under the laws of the USA. Given this, you need to be very careful in securing rights and fulfilling requirements for another's intellectual property. Respect the authors who allowed us to use our work, due to which we accelerated, optimized solutions, solved our problems and ultimately provided quality service to our customers. Let's comply with the rights and requirements. This is the first task.
- And the second?
- The second task is information security. It is clear that most licenses contain a disclaimer stating that the author / developer / contributor is not responsible for the possible harm that will be caused by the operation of this software. This is right, this is a responsibility that transfers to the consumer and requires maturity from him. Everything is not free.
You must pay for this responsibility and, of course, work with these risks. Not all companies can do this. We have a very strong department of information security - we are lucky. Therefore, we are serious about the presence of vulnerabilities and malicious code in general. Everyone who plans to use Open Source must take into account all the risks - not only legal, but also in terms of information security.
- What licenses do you like?
- Academic.
- O! Let's be more specific. There is MIT / BSD, etc., and there are virus copyleft licenses like Affero GPL. Which of these?
- Oh well. You cannot love or dislike a license. The product is made for a specific task and will be used in a specific way. When using open source, your task is to make sure that you provide your product or service without violating the rights of third parties. In this case, of course, you can use copylefts (for example, the GPL), if you ensure their use so that they do not violate restrictions and anyone else's rights. Of course, there are fewer difficulties when using academic licenses, simply because they carry fewer restrictions and therefore are easier to follow. For short, I call “academic” MIT, BSD, Apache, etc.
- Okay, do ordinary developers have to deal with information security? Or is it allocated to a separate department?
- All developers must understand the basics of information security and the principles of its security for their systems. But in our case, we work with individual developers who specialize in threats to information security. Moreover, we turn to them not only for the analysis of open source products, but also for the analysis of algorithms and design solutions.
“It is clear that these special security guards know everything.” And what does an ordinary developer need to know in this regard? What are the basic points?
- Model of threats, protection of channels, data protection. What is prone to threats: maybe this is a user interface or data transmission over a network (everything is distributed with us, so this is a very important issue). Basic tools like encryption, SSL, TLS, authentication, authorization, token handling and so on. You don’t need to know much.
- Rumor has it that you have anything to do with Apache Ignite :-)
- In terms of contributing, this is the main project that I am currently working on. Participation in Apache Ignite belongs to my second task - to ensure a balanced investment in Free and Open Source projects. This implies both the competent use of products (it is clear that the use of libraries is an investment, we, as users, increase the attractiveness of the product), as well as development, contributing.
For me, probably, this task is even more significant. We pay tribute to those products that we use in our company, and thanks to which we built a lot of products and systems. We try to improve them and ensure the possibility of use in companies such as ours: to optimize, bring to an enterprise state.
Apache Ignite is not the only project, but we will very intensively smuggle it into it, because one of the key platforms in the bank is being built on the basis of Apache Ignite. Ignite is a distributed computing grid that allows you to store and process data in memory, and in fact it is the basis of the IT landscape of our business. Therefore, we are extremely interested in its development.
- Many people know that Sbertech uses GridGain, and you're talking about Apache Ignite. What is the difference between the two?
- GridGain is an open core product built on the basis of an open core, which is Ignite. And GridGain is a set of plug-ins for this core, which simplify maintenance and operation procedures, provide a number of important information security and reliability requirements. But, in fact, the core is the most significant part, and plugins allow you to exploit all this in a real enterprise. And the bank already operates GridGain.
- Since Ignite is open, you can talk about it a bit, right? Do you only exploit it or do something to finish it, interact with developers?
- We intensively modify it. Directions of tasks are clearly defined, for example, ensuring performance taking into account the specifics of Sberbank: large-scale, ocean of data, high operational activity. Therefore, it should be fast and a lot. By this I mean both latency and throughput.
The second is to ensure reliability, i.e. availability and fault tolerance.
Third is operational efficiency, TCO management. Given the size of Sberbank, even a slight reduction in the use of resources, for example, disks by a certain percentage, on our scales gives tremendous savings.
And the fourth is the task of functional development. In fact, the main thing is the development of interfaces and integration with other components of the Sberbank technological stack. This is useful and important in terms of building a mature and integrated IT architecture.
Separately, there is the task of eliminating technical debt and defects (which always exist). It can probably be attributed to reliability.
- Let's go over these points for clarification. You say that you are working to improve performance, latency, throughput, that's all. The question is - does Ignite have any problems with this? I mean, is there something to modify or is it an ideal product?
- No, the product cannot be considered ideal. In each release, we drive both general benchmarks and microbenchmarks on specific components, we are constantly working on performance - we should not stop here. The task is difficult, because the components and solutions are already quite tightened up, the performance is almost at the limit of iron. This adds complexity, but there is always room for improvement. We have different use cases, new user tasks appear, in which there is the possibility of optimization. For example, optimize the tape drive for the specific nature of the data. There are tasks to optimize the network layer, which, again, depends on individual cases. Therefore, you always need to keep your finger on the pulse.
“You said that you would contribute back to the community.” And all these decisions about various cases and optimization for them are some kind of tradeoff. When we take our tradeoffs and bring them into the community, it may turn out that people in the community have slightly different conditions, different priorities. How to organize interaction and still copy the code that is needed for your cases?
- Other customers with other tasks. It is absolutely true that this is a standard problem. It all depends on the architecture of the solution. If the solution includes, for example, the ability to make plug-in extensions, plug-ins, libraries for different user solutions - you can get out. For example, if there is a comparator, then the user can always develop a solution that will pass this comparator to the input, and this will solve the problem based on specific conditions. Once again, capabilities are very architecture dependent. It’s wrong to simply roughly code and sculpt for our task without thinking about other clients - such pull requests do not go through a review.
Everyone understands what an Open Source project is, and in general, you can influence it. Of course, there are communities in which corporations are clearly present that influence development in their own interests, but if we play pure Open Source, then it will be correctly compared with meritocracy (the authority of the worthy). Prove that your decision is good, and then it will be made. Acting, as is often the case in closed development, that is, from the position of "I am the boss, I said so" will not work.
- One of the most interesting cases that Sbertech told in public is the Single Semantic Layer. A huge amount of data spread over an in-memory grid. How has this affected the open part of Ignite and how interesting are these developments to the community?
- Yes, such developments are underway, and we are very intensively working on tasks to ensure scalability and accessibility. We found cases in which the current topology management scheme is not optimal, because its temporal complexity grows from the number of nodes not quite as we would like. This somewhat complicates the achievement of the goal.
- As far as I remember, the cluster architecture is a ring. That is, when we join the ring, then in the beginning we go to the coordinator and then we go along the ring until we find the tail. And the more elements, the more time, right?
“Yes, sort of.” At the same time, with an increase in the number of nodes in the topology, both the size of the messages that are transmitted along the ring and the number of transitions between the participants increase. This is not to say that a ring is a bad decision, but in some cases it does not fit. Therefore, since the end of 2017, we in the community are finalizing topology management so that users can choose a topology management scheme: a ring (sometimes it fits perfectly) or a star on Zookeeper.
- And where did the ring come from? Why is it? Where is it perfect?
- This is a wonderful solution on the topologies of 100-200 nodes in one data center. Allows you to simply and reliably synchronize all nodes, they just go in order. If we go to the star, then they begin to work in parallel, faster, but at the same time synchronizing them becomes much more difficult. That is, the ring can be more stable and reliable, agree?
- Oh sure. But can it be done so that this topology can be changed by some parameter in the config, how is the setting?
- Yes, we are doing it now, we include both topologies in the release. Probably, the already proposed implementations do not cover all cases of users, and as soon as new ones appear, we will try to ensure their effective processing.
- As far as I understand, this is a rather complicated revision. And this revision is done by people in Sbertekh, during working hours, or in the evenings for pleasure?
- This is done by the community, which includes PBT employees, whose main task during working hours is to contribute to this project. The topology problem affects one of the key solutions in the product core, so the main burden fell on DiscoverySPI maintainers, but I hope that the participation of our developers also positively affected the result.
- Well, that is, these are people who solve a problem during working hours, but at the same time are members of the community.
- Yes, the most significant part of the work of our developers takes place in the community. But I also see from our guys such commits that appeared in an hour, two, three nights.
- And these employees will not have a problem from the fact that they work in a bank, on a closed system, and at the same time commit to an open source?
- No, it will not. All participants are official corporate contributors. The creation of the direction and the decision on investments were made at the level of the company’s management, and yes, this group of dedicated corporate contributors who, in accordance with all the company’s and TC standards, develop Open Source products in the interests of the company. Yes - this is development and Open Source, yes - this is during working hours, and during non-working hours, too, but this is already if the community asks for it.
- We just talked about some external affairs that the community decides. But most likely, the company needs to make its own integrations, improve for some of its cases ... Have you written a lot of your own? Or is it just a little dopilka?
- Speaking about the Apache Ignite project, over the last quarter, our contribution to the project amounted to 8-10 percent of all changes, and we strive to increase this percentage. We wrote a lot, and this is not only the development and optimization of the existing functionality, we are also working on new functionality for the project. This is a challenge for the community, and responsibility for us, since after its inclusion, the community to some extent has the task of supporting it.
Tasks can appear not only from the community, but also from users within the company: architects, development and maintenance teams. The development of the project on these tasks also significantly affects the product.
- But, let’s say, there were several reports from the Sbertekh program of the PRPRB regarding its “special feng shui”. Do I need to write any additional tools and admin pages to support this?
- Interfaces for operation are constantly evolving. The management console of the same Oracle is more familiar to maintain and has more functionality. Whether it needs to be fully reproduced is another question.
- And in open form, you can see the management console?
- Oh sure. Web Console published, Visor, CLI - everything is public.
- And if you look at it more globally, what are the general directions and goals?
- Now we are more focused on the development of Apache Ignite, which meets the company's priorities. But our technological stack does not end there. We work with many Open Source projects, where we see opportunities for development, and we have something to offer to improve these projects. I hope that they will be interesting not only to us, but also to other users. For each project, we determine the necessary volume of our participation (from fixing defects to changing the architecture), evaluate the presence of the necessary competencies, the willingness and interest of the community in cooperation. As a result of all this, we understand the volume of required resources on our part. For the project, the value of our participation in the fact that we can offer cases of a very large bank. This can give a project a serious impetus for growth. We already have such cases.
“You said that you can bring usages of a large bank.” How will these user cases differ from the user cases of something else?
- The main difference between Sberbank and the rest is reliability requirements, i.e. availability, durability, fault tolerance, etc.
- And security, I guess.
- Yes, security is a separate issue. We hope that you can bring the Open Source project and go the line of adaptation to these requirements. Otherwise, he will be looking for usages for a very long time. Not everyone is faced with the requirements that are in the bank.
- And are there any popular, by everyone's ears, products that are still sawing and sawing to such a scale? For example, some distributed file storage, ceph?
“Well, yes, ceph is a good example.” The project is good, wonderful, very mature, but it can still be improved.
- Do you use any virtual developers inside for developers? What is it controlled by?
- OpenStack.
- As far as I understand, OpenStack is such a thing that can really be modified. Do you do anything with it, or how did the vanilla put it, and it works?
- We have not yet finalized OpenStack, but definitely, this is an interesting direction, like Cloud Foundry, containers.
“What about the containers?”
- We have a wonderful experience with them :-) We understand that on our scale, to ensure efficient utilization and resource management, we must implement (and are implementing now) containerization of applications. Here the question also arises of involving these projects in the development, because increasing competitive advantages is a task that is useful both to us and to the project itself.
- Let's talk about people.
- Given the ambitions and current expectations of the bank, at some point we will turn into one of the largest corporate contributors in Open Source and will be visible at the global level.
But there is a problem that there are few strong system developers in the market. And in the company, too, since the bank has always concentrated more on applied development, development of business applications.
- That is, the finished technologies were glued in a special way and specific business tasks were solved?
- Yes, rather, business tasks were solved: granting a loan, scoring calculation, etc. Of course, we have made and are making unique decisions in terms of customer experience, performance, reliability, security. But we are approaching the limit of solutions offered by vendors and may face competitors approaching. Therefore, now we are talking about the development of our own platform, the intensive search and use of innovations in Open Source projects, which, in turn, requires expertise in system development.
In our country there is a good base in terms of student training. They are being prepared for what is needed in system development. They are not prepared to write business processes, they are prepared to seek and evaluate solutions, and they provide excellent mathematical and algorithmic training. This knowledge and skills are extremely important to us now. In the problems that we are now solving, a lot of things that would be of secondary importance in applied development come to the fore, mainly questions of the effectiveness of solutions.
But, unfortunately, since system development has not been in demand for a long time (with the exception of large IT companies such as Kaspersky, Mail, Yandex) - we will grow for a long time to the scale of at least hundreds of contributors. There are very few developers who can effectively engage in system development, although if you have a good academic base, you can build on the necessary skills.
The second is experience and deep knowledge of programming languages.
Thirdly, the requirements for sociability, because we work in a community. There is no boss who says “do it” to the developer; there is a community here that says “maybe I will accept” or “rather, I will not accept”. The problem is different, and we rebuilt under it. No one requires code from the developer, he himself must convince the community to accept his work, his contribution. It's much harder to accept criticism, communicate, review, explain and be public. Being helpful to other members of the community is a basic requirement that we make.
- And what motivation should a person have to deal with all this? Many come to work, work from eight to five with a compulsory lunch according to the instructions of the boss ... and everything that you described is not very good for this scheme.
- Motivation in development. We look closely at the candidates, trying to determine whether they understand themselves, and how well they understand where they want to come and why we need them. We provide developers with the opportunity to develop, have a public reputation at the global level, which does not nullify when moving to another employer (and such a reputation is much more valuable). The values of such developers are naturally aligned with our team and company, they understand why they want to work with us, and not just spend working time in the office.
But there are very, very few who can work effectively in an open community.
Work in the community and the development of Open Source differ from the work of other developers of the company. At our contributors, the processes are more integrated with the community, we primarily look at it. And only then we are already trying to rebuild our internal corporate processes to the expectations of the community. If others solve the problems of internal bureaucracy, then we solve the problems of interaction with the community.
“Why are you doing this?” What is your motivation?
- For me, this is a challenge and an opportunity to positively influence Sberbank’s IT and increase its competitiveness. But, which is also extremely important, is the opportunity to influence the development of IT in general by contributing to open projects. Of course, all this is connected with the main commercial goals of the company - brand development, increasing ROI, building the Platform, etc., therefore this direction is in demand by the bank's management.
- In general, somehow in Russia, system programming is not very popular. Often I hear people say: if I get down to system programming, then as a result, I won’t find work anywhere. It’s good for everyone to create websites, to know some basic things like Spring and Hibernate, and if I get to learn multithreading, then this is awesome, no one in this country needs to, but still have to go abroad. And an activity like yours can, at a minimum, increase the popularity of these competencies. If only because with them you can go to you.
- Perhaps that is why there are more good applicants on the market than system developers. I am very glad that we managed to assemble the current team, where all the guys are very bright. It is amazing how they think and work, they can solve almost any problem. I hope that we can attract even more talented developers. Therefore, one cannot say for sure that in Russia no one needs system developers.
Over time, Sbertech has been increasingly active in the OSS community, and we decided to ask them about it. How does strict banking specificity combine with the open source spirit of freedom? What are the requirements for Open Source that other companies might not have? Are there any employees in Sbertech who write open source as their main work tasks? What are your plans and desires for the future? Anton Churaev , who oversees Free & Open Source, told us about all this and more.
Oleg: Hi Anton. Let's introduce you to the Khabrovites a little. Tell about yourself: who are you, what are you doing?
Anton: I am an engineer who, however, develops only in his spare time. Now I am building in Sbertech practices and competencies for the development and application of Free & Open Source products. You need to understand that these are slightly different things.
- Yes, I understand, a couple of times when talking with Stallman I called Free as Open and after that I listened to such a lecture that I remembered for a lifetime :-) Well, what is your position?
- Curator of the development of Free & Open Source. And open source enthusiast :)
- Can you decode a little more about the "competencies for the implementation of Open Source"? It sounds like some kind of secret knowledge.
- Few people imagine what Open Source means for corporations. On the one hand, these are innovations and the absence of the need to develop commodity, focus on the competitive advantages of their products, reuse and reduce costs. Often these are projects that have actually become industry standards. Take the same Hadoop - everyone knows it, everyone knows it, it has long been a standard. Or the most common databases - the top five are three open source products - MySQL, PostgreSQL and MongoDB.
But few people think that using OpenSource involves a lot of hidden costs. This is not to say that we found something open source and solved all our problems. For example, there are big questions about “legal hygiene”. When working with vendor software, everything is very clear: I took a license, you work on it and use support. When working with Open Source, much is left at the mercy of developers and users. In this case, legal and legal issues are one of the first places. In addition, in Russia there are nuances. If all over the world the concept of intellectual property is quite developed and everyone understands that it is very important that there is a specific owner, then historically in Russia everything turned out differently. Here we are not so careful and respectful of intellectual property, although this is extremely wrong.
- Can I clarify? What is the legal status of GPL licenses in Russia? For example, the GPL does not allow modifications to local laws and does not indicate territorial restrictions. Therefore, such an agreement is not compatible with the legal regime established in the territory of the Russian Federation, and this is very, very bad.
- I do not want to divide licenses into some zones. Sberbank is a global company, so the software can be used both in the United States and in the European Union. And, as I understand it (I am not a lawyer, but to the best of my knowledge), in case of violation of the restrictions of the licenses of the copyright holder in the territory of, for example, the USA, we will be responsible under the laws of the USA. Given this, you need to be very careful in securing rights and fulfilling requirements for another's intellectual property. Respect the authors who allowed us to use our work, due to which we accelerated, optimized solutions, solved our problems and ultimately provided quality service to our customers. Let's comply with the rights and requirements. This is the first task.
- And the second?
- The second task is information security. It is clear that most licenses contain a disclaimer stating that the author / developer / contributor is not responsible for the possible harm that will be caused by the operation of this software. This is right, this is a responsibility that transfers to the consumer and requires maturity from him. Everything is not free.
You must pay for this responsibility and, of course, work with these risks. Not all companies can do this. We have a very strong department of information security - we are lucky. Therefore, we are serious about the presence of vulnerabilities and malicious code in general. Everyone who plans to use Open Source must take into account all the risks - not only legal, but also in terms of information security.
- What licenses do you like?
- Academic.
- O! Let's be more specific. There is MIT / BSD, etc., and there are virus copyleft licenses like Affero GPL. Which of these?
- Oh well. You cannot love or dislike a license. The product is made for a specific task and will be used in a specific way. When using open source, your task is to make sure that you provide your product or service without violating the rights of third parties. In this case, of course, you can use copylefts (for example, the GPL), if you ensure their use so that they do not violate restrictions and anyone else's rights. Of course, there are fewer difficulties when using academic licenses, simply because they carry fewer restrictions and therefore are easier to follow. For short, I call “academic” MIT, BSD, Apache, etc.
- Okay, do ordinary developers have to deal with information security? Or is it allocated to a separate department?
- All developers must understand the basics of information security and the principles of its security for their systems. But in our case, we work with individual developers who specialize in threats to information security. Moreover, we turn to them not only for the analysis of open source products, but also for the analysis of algorithms and design solutions.
“It is clear that these special security guards know everything.” And what does an ordinary developer need to know in this regard? What are the basic points?
- Model of threats, protection of channels, data protection. What is prone to threats: maybe this is a user interface or data transmission over a network (everything is distributed with us, so this is a very important issue). Basic tools like encryption, SSL, TLS, authentication, authorization, token handling and so on. You don’t need to know much.
- Rumor has it that you have anything to do with Apache Ignite :-)
- In terms of contributing, this is the main project that I am currently working on. Participation in Apache Ignite belongs to my second task - to ensure a balanced investment in Free and Open Source projects. This implies both the competent use of products (it is clear that the use of libraries is an investment, we, as users, increase the attractiveness of the product), as well as development, contributing.
For me, probably, this task is even more significant. We pay tribute to those products that we use in our company, and thanks to which we built a lot of products and systems. We try to improve them and ensure the possibility of use in companies such as ours: to optimize, bring to an enterprise state.
Apache Ignite is not the only project, but we will very intensively smuggle it into it, because one of the key platforms in the bank is being built on the basis of Apache Ignite. Ignite is a distributed computing grid that allows you to store and process data in memory, and in fact it is the basis of the IT landscape of our business. Therefore, we are extremely interested in its development.
- Many people know that Sbertech uses GridGain, and you're talking about Apache Ignite. What is the difference between the two?
- GridGain is an open core product built on the basis of an open core, which is Ignite. And GridGain is a set of plug-ins for this core, which simplify maintenance and operation procedures, provide a number of important information security and reliability requirements. But, in fact, the core is the most significant part, and plugins allow you to exploit all this in a real enterprise. And the bank already operates GridGain.
- Since Ignite is open, you can talk about it a bit, right? Do you only exploit it or do something to finish it, interact with developers?
- We intensively modify it. Directions of tasks are clearly defined, for example, ensuring performance taking into account the specifics of Sberbank: large-scale, ocean of data, high operational activity. Therefore, it should be fast and a lot. By this I mean both latency and throughput.
The second is to ensure reliability, i.e. availability and fault tolerance.
Third is operational efficiency, TCO management. Given the size of Sberbank, even a slight reduction in the use of resources, for example, disks by a certain percentage, on our scales gives tremendous savings.
And the fourth is the task of functional development. In fact, the main thing is the development of interfaces and integration with other components of the Sberbank technological stack. This is useful and important in terms of building a mature and integrated IT architecture.
Separately, there is the task of eliminating technical debt and defects (which always exist). It can probably be attributed to reliability.
- Let's go over these points for clarification. You say that you are working to improve performance, latency, throughput, that's all. The question is - does Ignite have any problems with this? I mean, is there something to modify or is it an ideal product?
- No, the product cannot be considered ideal. In each release, we drive both general benchmarks and microbenchmarks on specific components, we are constantly working on performance - we should not stop here. The task is difficult, because the components and solutions are already quite tightened up, the performance is almost at the limit of iron. This adds complexity, but there is always room for improvement. We have different use cases, new user tasks appear, in which there is the possibility of optimization. For example, optimize the tape drive for the specific nature of the data. There are tasks to optimize the network layer, which, again, depends on individual cases. Therefore, you always need to keep your finger on the pulse.
“You said that you would contribute back to the community.” And all these decisions about various cases and optimization for them are some kind of tradeoff. When we take our tradeoffs and bring them into the community, it may turn out that people in the community have slightly different conditions, different priorities. How to organize interaction and still copy the code that is needed for your cases?
- Other customers with other tasks. It is absolutely true that this is a standard problem. It all depends on the architecture of the solution. If the solution includes, for example, the ability to make plug-in extensions, plug-ins, libraries for different user solutions - you can get out. For example, if there is a comparator, then the user can always develop a solution that will pass this comparator to the input, and this will solve the problem based on specific conditions. Once again, capabilities are very architecture dependent. It’s wrong to simply roughly code and sculpt for our task without thinking about other clients - such pull requests do not go through a review.
Everyone understands what an Open Source project is, and in general, you can influence it. Of course, there are communities in which corporations are clearly present that influence development in their own interests, but if we play pure Open Source, then it will be correctly compared with meritocracy (the authority of the worthy). Prove that your decision is good, and then it will be made. Acting, as is often the case in closed development, that is, from the position of "I am the boss, I said so" will not work.
- One of the most interesting cases that Sbertech told in public is the Single Semantic Layer. A huge amount of data spread over an in-memory grid. How has this affected the open part of Ignite and how interesting are these developments to the community?
- Yes, such developments are underway, and we are very intensively working on tasks to ensure scalability and accessibility. We found cases in which the current topology management scheme is not optimal, because its temporal complexity grows from the number of nodes not quite as we would like. This somewhat complicates the achievement of the goal.
- As far as I remember, the cluster architecture is a ring. That is, when we join the ring, then in the beginning we go to the coordinator and then we go along the ring until we find the tail. And the more elements, the more time, right?
“Yes, sort of.” At the same time, with an increase in the number of nodes in the topology, both the size of the messages that are transmitted along the ring and the number of transitions between the participants increase. This is not to say that a ring is a bad decision, but in some cases it does not fit. Therefore, since the end of 2017, we in the community are finalizing topology management so that users can choose a topology management scheme: a ring (sometimes it fits perfectly) or a star on Zookeeper.
- And where did the ring come from? Why is it? Where is it perfect?
- This is a wonderful solution on the topologies of 100-200 nodes in one data center. Allows you to simply and reliably synchronize all nodes, they just go in order. If we go to the star, then they begin to work in parallel, faster, but at the same time synchronizing them becomes much more difficult. That is, the ring can be more stable and reliable, agree?
- Oh sure. But can it be done so that this topology can be changed by some parameter in the config, how is the setting?
- Yes, we are doing it now, we include both topologies in the release. Probably, the already proposed implementations do not cover all cases of users, and as soon as new ones appear, we will try to ensure their effective processing.
- As far as I understand, this is a rather complicated revision. And this revision is done by people in Sbertekh, during working hours, or in the evenings for pleasure?
- This is done by the community, which includes PBT employees, whose main task during working hours is to contribute to this project. The topology problem affects one of the key solutions in the product core, so the main burden fell on DiscoverySPI maintainers, but I hope that the participation of our developers also positively affected the result.
- Well, that is, these are people who solve a problem during working hours, but at the same time are members of the community.
- Yes, the most significant part of the work of our developers takes place in the community. But I also see from our guys such commits that appeared in an hour, two, three nights.
- And these employees will not have a problem from the fact that they work in a bank, on a closed system, and at the same time commit to an open source?
- No, it will not. All participants are official corporate contributors. The creation of the direction and the decision on investments were made at the level of the company’s management, and yes, this group of dedicated corporate contributors who, in accordance with all the company’s and TC standards, develop Open Source products in the interests of the company. Yes - this is development and Open Source, yes - this is during working hours, and during non-working hours, too, but this is already if the community asks for it.
- We just talked about some external affairs that the community decides. But most likely, the company needs to make its own integrations, improve for some of its cases ... Have you written a lot of your own? Or is it just a little dopilka?
- Speaking about the Apache Ignite project, over the last quarter, our contribution to the project amounted to 8-10 percent of all changes, and we strive to increase this percentage. We wrote a lot, and this is not only the development and optimization of the existing functionality, we are also working on new functionality for the project. This is a challenge for the community, and responsibility for us, since after its inclusion, the community to some extent has the task of supporting it.
Tasks can appear not only from the community, but also from users within the company: architects, development and maintenance teams. The development of the project on these tasks also significantly affects the product.
- But, let’s say, there were several reports from the Sbertekh program of the PRPRB regarding its “special feng shui”. Do I need to write any additional tools and admin pages to support this?
- Interfaces for operation are constantly evolving. The management console of the same Oracle is more familiar to maintain and has more functionality. Whether it needs to be fully reproduced is another question.
- And in open form, you can see the management console?
- Oh sure. Web Console published, Visor, CLI - everything is public.
- And if you look at it more globally, what are the general directions and goals?
- Now we are more focused on the development of Apache Ignite, which meets the company's priorities. But our technological stack does not end there. We work with many Open Source projects, where we see opportunities for development, and we have something to offer to improve these projects. I hope that they will be interesting not only to us, but also to other users. For each project, we determine the necessary volume of our participation (from fixing defects to changing the architecture), evaluate the presence of the necessary competencies, the willingness and interest of the community in cooperation. As a result of all this, we understand the volume of required resources on our part. For the project, the value of our participation in the fact that we can offer cases of a very large bank. This can give a project a serious impetus for growth. We already have such cases.
“You said that you can bring usages of a large bank.” How will these user cases differ from the user cases of something else?
- The main difference between Sberbank and the rest is reliability requirements, i.e. availability, durability, fault tolerance, etc.
- And security, I guess.
- Yes, security is a separate issue. We hope that you can bring the Open Source project and go the line of adaptation to these requirements. Otherwise, he will be looking for usages for a very long time. Not everyone is faced with the requirements that are in the bank.
- And are there any popular, by everyone's ears, products that are still sawing and sawing to such a scale? For example, some distributed file storage, ceph?
“Well, yes, ceph is a good example.” The project is good, wonderful, very mature, but it can still be improved.
- Do you use any virtual developers inside for developers? What is it controlled by?
- OpenStack.
- As far as I understand, OpenStack is such a thing that can really be modified. Do you do anything with it, or how did the vanilla put it, and it works?
- We have not yet finalized OpenStack, but definitely, this is an interesting direction, like Cloud Foundry, containers.
“What about the containers?”
- We have a wonderful experience with them :-) We understand that on our scale, to ensure efficient utilization and resource management, we must implement (and are implementing now) containerization of applications. Here the question also arises of involving these projects in the development, because increasing competitive advantages is a task that is useful both to us and to the project itself.
- Let's talk about people.
- Given the ambitions and current expectations of the bank, at some point we will turn into one of the largest corporate contributors in Open Source and will be visible at the global level.
But there is a problem that there are few strong system developers in the market. And in the company, too, since the bank has always concentrated more on applied development, development of business applications.
- That is, the finished technologies were glued in a special way and specific business tasks were solved?
- Yes, rather, business tasks were solved: granting a loan, scoring calculation, etc. Of course, we have made and are making unique decisions in terms of customer experience, performance, reliability, security. But we are approaching the limit of solutions offered by vendors and may face competitors approaching. Therefore, now we are talking about the development of our own platform, the intensive search and use of innovations in Open Source projects, which, in turn, requires expertise in system development.
In our country there is a good base in terms of student training. They are being prepared for what is needed in system development. They are not prepared to write business processes, they are prepared to seek and evaluate solutions, and they provide excellent mathematical and algorithmic training. This knowledge and skills are extremely important to us now. In the problems that we are now solving, a lot of things that would be of secondary importance in applied development come to the fore, mainly questions of the effectiveness of solutions.
But, unfortunately, since system development has not been in demand for a long time (with the exception of large IT companies such as Kaspersky, Mail, Yandex) - we will grow for a long time to the scale of at least hundreds of contributors. There are very few developers who can effectively engage in system development, although if you have a good academic base, you can build on the necessary skills.
The second is experience and deep knowledge of programming languages.
Thirdly, the requirements for sociability, because we work in a community. There is no boss who says “do it” to the developer; there is a community here that says “maybe I will accept” or “rather, I will not accept”. The problem is different, and we rebuilt under it. No one requires code from the developer, he himself must convince the community to accept his work, his contribution. It's much harder to accept criticism, communicate, review, explain and be public. Being helpful to other members of the community is a basic requirement that we make.
- And what motivation should a person have to deal with all this? Many come to work, work from eight to five with a compulsory lunch according to the instructions of the boss ... and everything that you described is not very good for this scheme.
- Motivation in development. We look closely at the candidates, trying to determine whether they understand themselves, and how well they understand where they want to come and why we need them. We provide developers with the opportunity to develop, have a public reputation at the global level, which does not nullify when moving to another employer (and such a reputation is much more valuable). The values of such developers are naturally aligned with our team and company, they understand why they want to work with us, and not just spend working time in the office.
But there are very, very few who can work effectively in an open community.
Work in the community and the development of Open Source differ from the work of other developers of the company. At our contributors, the processes are more integrated with the community, we primarily look at it. And only then we are already trying to rebuild our internal corporate processes to the expectations of the community. If others solve the problems of internal bureaucracy, then we solve the problems of interaction with the community.
“Why are you doing this?” What is your motivation?
- For me, this is a challenge and an opportunity to positively influence Sberbank’s IT and increase its competitiveness. But, which is also extremely important, is the opportunity to influence the development of IT in general by contributing to open projects. Of course, all this is connected with the main commercial goals of the company - brand development, increasing ROI, building the Platform, etc., therefore this direction is in demand by the bank's management.
- In general, somehow in Russia, system programming is not very popular. Often I hear people say: if I get down to system programming, then as a result, I won’t find work anywhere. It’s good for everyone to create websites, to know some basic things like Spring and Hibernate, and if I get to learn multithreading, then this is awesome, no one in this country needs to, but still have to go abroad. And an activity like yours can, at a minimum, increase the popularity of these competencies. If only because with them you can go to you.
- Perhaps that is why there are more good applicants on the market than system developers. I am very glad that we managed to assemble the current team, where all the guys are very bright. It is amazing how they think and work, they can solve almost any problem. I hope that we can attract even more talented developers. Therefore, one cannot say for sure that in Russia no one needs system developers.