“They did microservices before it became mainstream”: Sberbank-Technologies about development

Sberbank Technologies is engaged in a number of interesting tasks - from unifying the interfaces of a huge bank to prototyping blockchain. In most cases, development is conducted in Java: the company considers it the best choice for large enterprise systems, appreciating the ability to build a solution from third-party products and internal developments.

At conferences, the speakers of Sbertekh talk both directly about their experience and about the general conclusions that can be drawn from it. But not everything fits into the format of reports, and on the eve of Joker we separately asked questions to three employees of the company speaking at Java-conferences. We got a reply:

• Sergey vlsergey Vladimirov (will perform at Joker 2016, previously performed at JPoint)
• Alexander Matorin (previously performed at Joker, JPoint and JBreak)
• Maxim Zelinsky (previously performed at JPoint and JBreak)

Sergey Vladimirov (head of development in the Central Committee of a platform for supporting business development)

- What are you working on at Sbertech?

- For about a year I was developing a new system of accounting for card acceptance terminals at retail outlets. The project replaces an outdated system written in C, working with a direct connection to the database and having a large number of manual and semi-manual interactions with other systems. Our application will be part of the new platform of Sberbank 2018+ (target platform from 2018). It is written as an “honest” server-side Java application (with a browser as a client), using a client-oriented data storage architecture (based on Apache Ignite / GridGain technology) and a single corporate data model. There are a lot of buzzwords here, but behind each of them a cardinal change, if not in how to think, then certainly in how to develop business applications.

The first phase was completed in this project, my colleagues will continue to work on it, and I myself am starting to engage in a new project for organizing the user interface. Again, this will not be just a rewrite of the code from HTML to JavaScript, but the introduction of new approaches to the interaction of frontend and backend.

- Do you teach information security and cryptography - is this related to working in Sbertekh, or are you interested in them separately? Sberbank began to prototype blockchain - are you related to this?

- I started to get involved in information security since school (as well as programming), but it always went in parallel and was not directly connected. But they exert influence on each other constantly. At a minimum, you find yourself starting to look at the same problem from two sides: “how I would do it as a developer” and “where this system has weaknesses if I want to break it”. This is especially true for complex enterprise level systems. It’s very bad if “security considerations” in the design of systems are that a ready-made solution is forced to change under the pressure of security personnel, and not vice versa - when following the safety rules is laid down in the principles of building the system, and the one who develops the system understands and “ feels ”these are principles. Very useful thing.

Not to mention the fact that I as a programmer cannot be scared by the scary words “certificate”, “digital signature” or “hash sum”.

As for the blockchain, unfortunately (I'm really sorry), Sberbank has so many interesting projects that it’s not just impossible to manage to participate in all those interesting, but sometimes you even learn about them from the news and the media. About the same blockchain, I was pleased to attend a lecture organized by Sberbank for my employees, I learned what blockchain is from a business point of view, but, apparently, I will not have time to personally participate in this project in the coming year. Unfortunately, I don’t have time to take part in the “Data Factory” project (the big data analysis project).

- Do you like open source - how actively do you use it when working in Sbertekh?

- You may not like open source, but it is very difficult not to use it :-) If you are a Java programmer and are developing a solution for business, it means that you have more than a dozen open source libraries in the classpath, ranging from the logger to XML libraries or JSON In Sberbank Technologies, we not only use open source, but also try to be an active part of the community. The most striking example is, of course, participating in work on Apache Ignite.

Open source is not just cheaper. It turns out to be faster in terms of “time to market”. Using a paid product with closed source code, you can wait two or three years from the developer for corrections or improvements to the functionality you need. Or you can take an open source solution, fix some flaws in it (if they were there and nobody noticed them), and when you need to add or change something, just take and change. Let this require qualifications, even if it is a little unusual for your employees, but it will work, roughly speaking, after a week in production, and not after 2-3 years, when a new version of a closed product is released. In Sberbank Technologies, this is not a theory, but real cases when they are forced to change some technology or solution simply because it is too expensive to modify an existing one, or, which is usually more important,

- You speak to the public in different formats: both as a teacher at the Moscow Institute of Physics and Technology, and with scientific lectures, and at conferences. How is the conference format different for you?

- If you compare lectures and speeches at conferences, the latter are always more difficult and interesting. The speaker has no way to get people to come to his story - he will not put a deuce on the exam. You can’t just go out to the audience and tell the material - people didn’t come to take notes, not just learn new and informative things, but people came to get the knowledge that will help them become professionals in their field. And if some report does not provide something useful that a person can put into practice (even in the future), the person will get up and leave.

That is, the student’s goal is to pass the exam. The goal of a student of a popular science lecture is to learn something interesting in their free time. Go to a lecture as an excursion. But the goal of a professional conference attendee is professional (and career) growth. And if the report does not contribute to this goal, they will not listen to it.

- What will you talk about at the upcoming Joker?

- The presentation for Joker was refined for a long time with the organizers, there were several preliminary auditions, and in the end we decided to shift the emphasis a bit. It was previously assumed that there would be several examples of optimizations at the algorithm level, when with simple actions (and definitely without new libraries and tools) we get a significant performance gain. But in the end, we decided to take one of the highlights and consider it from several angles.

Therefore, on the student edition, you will find a story about such a data structure as trie, or a loaded (prefix) tree. And a real example from working experience will be shown in which, with the help of this structure, the productivity of a single narrow task has been increased a thousand times.

Alexander Matorin (Chief Development Officer at the Central Committee for the Development of Risk Systems)

- You are engaged in a system for detecting fraud, and immediately curious: can you tell us more (without giving out any secrets that would help scammers)?

I work in a department that develops risk management systems in the financial markets. There are systems that collect transactions made by traders on behalf of Sberbank. Other systems may subscribe to receive new trade events in order to calculate certain risk metrics, or check the transaction for fraud.

The simplest example of fraud tracking is the search for “paired deals”. That is, a trader buys a certain financial instrument and after a while sells it at a lower price, possibly to the same counterparty. For example, a trader bought dollars at 70 rubles, and a day later sold at 60, expecting that part of the difference will return to him by another channel.

Other fraud tests consider more sophisticated metrics, such as Implied Volatility for FX options using the Black-Scholes method. That is, there is an interesting subject area and a lot of financial mathematics.

- What tools / technologies do you use?

About 15 different systems are being developed in our department, which interact with each other using different protocols. The microservice architecture, which we started using about 4 years ago, even before it became mainstream, fits very well here. Calculation of each risk metric or transaction aggregator is a separate service. Now our systems are divided into approximately 100 microservices that run in several data centers. We write in Java 8, Spring is used everywhere. Different services use different technologies and bases. Part of what we use: Cassandra, MongoDB, Coherence, GridGain, RabbitMQ, Oracle DB, TomCat.

- You also participate in internal conferences of the company - can you tell us about them?

- For two consecutive years, Sberbank Technologies has held internal Dev & QA days conferences. About 150 people came to the last Dev Day in Moscow, about 300 watched the online broadcast. There were approximately 20 reports in 3 streams. I participated as a speaker and as a member of the program committee. Here, like at Joker, anyone could apply for a report. After the application is considered by the program committee. If the application is accepted, one or several rehearsals of the report are conducted with the speaker. I hope these conferences will develop further.

- Do you teach at the Sbertekh department at the Moscow Institute of Physics and Technology - but is this an additional activity for you that does not affect the core one, or do you feel that when teaching others, in the end you yourself begin to perform tasks better when working in Sbertekh?

- I have been teaching at the MBT magistracy at MIPT for more than 2 years. Also, a Java school was opened in SBT a couple of months ago, where I also lecture.

The main objective of my Java course, which I set for myself, is not just to ensure that students remember what I said, but to convey to them their passion for Java, so that they themselves deepen their expertise with interest. Each year, the department provides the best students with the opportunity to attend Joker and JPoint conferences.

In the second semester, I read a course on distributed systems. I tell CAP, FLP theorems, distributed transaction algorithms, consensus protocols, the concept of time in distributed systems, what can break and how to provide fault tolerance, NoSQL database architectures.

The second course helped me to learn a lot of new things myself, to structure my knowledge and apply some of this in my projects.

Algorithmic trading competitions have been held for students of the course for 2 years already. Students should implement algorithmic trading strategies on the Java interface. Then upload this source code to the game’s website. The strategy is compiled, launched in a separate JVM and in real time begins to trade with solutions from other participants. You can upload new versions of strategies an unlimited number of times. Last year, one of the participants downloaded approximately 160 versions by the end of the game. But another won :)

Maxim Zelinsky (department head in the Central Committee for the Development of Frontal Systems)

- What exactly are you doing?

- I participate in the creation of the Unified Front-End System (EFS), namely, I am responsible for the creation and development of the Platform for developers directly implementing business functionality within the EFS.

In the future, the Unified Front System will serve absolutely all Bank customers through all available channels (Internet banks, mobile applications, ATMs, call centers, branches, etc.). Its task is to unify and reuse solutions. Of course, different channels have their own specifics, but in general the essence of the program is to invest once in the implementation of something and make this implementation similar for different channels.

Given the enormous scale of ESF, the creation of high-quality and reliable solutions requires a single set of technologies and services to reduce the time to market of new solutions, as well as guarantee a uniform level of service in terms of reliability and performance. These issues (a single stack, services, architecture) are dealt with by my team within the framework of the ESF Platform.

- You specialize in things like distributed systems - why then do you deal with the front-end system, and not the backend?

- In our case, the front-end in terms of performance, fault tolerance and reliability is even more interesting than backing: there are requirements like “uptime 99.99%”. And if the backend does not work at some point, the client will not even notice it. His payments will not be executed instantly, but it will be possible to enter them into the system, it will be possible to obtain extracts from the cache.

But according to the backend, Sberbank Technologies has another big program: The Business Development Support Program. This is an attempt to transform technological solutions using the In-Memory Data Grid. At first there were attempts to make automatic sharding of data based on Oracle or PostgreSQL. There was a pilot, he gave both positive and negative results. According to the results of the pilot, it was decided "current technologies are not entirely relevant for the scale of the bank, let's look to the future." They chose IMDG, and colleagues are now moving back-end data arrays into memory.

- With such a gigantic scale as yours, are there situations when one technological solution leads to a very large gain in resources?

- A striking example is the choice of technologies for the web-presentation level of JS SPA and React. The mere transfer of presentation logic from the application server to the client allowed us to save ~ 200 application servers, which were originally planned to be used for the classic server-side web-presentation level. The same applies to reducing traffic between the client and server, which is obviously reduced due to the fact that the exchange is only data, not HTML.

- You have experience in reverse engineering - but is it practically useful when working in Sbertekh?

- Very much. Especially often you have to deal with reverse engeneering when working with IBM WebSphere Application Server, starting from IBM JVM, where the team I worked with before the ESF found a critical defect leading to deadlock inside the JVM when working with annotations under load, ending with Java drivers for IBM WebSphere MQ, when I had to study the problem for a long time, why the combination of IBM WAS + IBM MQ + Spring JMS Templates does not work with non-persistent queues under load.

- Thanks. We will wait for the report of Sergey Vladimirov at Joker, but for now we recall the previous reports from Sbertekh at Java conferences: