“It seems to me that we are approaching the time when everyone will be able to code. Code is freedom
Habr, hello! We are continuing a series of interviews with Newprolab alumni, in which they tell about their history of transition to the field of work with big data. The stories are all different and will be of interest to those who are thinking about changing their career trajectory or about how new knowledge can help to solve current problems. Today we have, most likely, the second option. Meet Dmitry Rodin, a developer at heart, a contributor to the open source analytical platform Rockstat ; co-founder, co-author and teacher of Digital God - educational programs for digital professionals; CEO Xeteq .
We talked with Dima about programming languages and what solutions he writes on them, about his open source project and why he does it, about the new specialty that they teach, etc.
- Dima, tell us a little about yourself.
- My caramel craft immediately comes to mind, I have long been joking with it, but seriously, I’m a programmer at heart, or an architect, I like to invent and create new solutions, to involve other people in them. For the last 10 years I have been doing marketing technologies, and I got so carried away with them that it turned into a hobby. I developed almost everything: from the advertising management system and DSP to the anti-fraud with hardware monitoring modules. I think I have known Zen, but this is not accurate.
- You have been programming for more than 15 years, tell me, how many programming languages do you know and use in your work and your personal projects?
- I think that counting the number of languages is wrong, the most important skill is to understand several concepts of languages, and then there will be practically no difference on what to write. Yes, of course, each language has its advantages and pitfalls, we should not forget about it.
- And from what languages did you start and what do you use now.
- Now for me there is no particularly big difference on what to write. I started with the development of sites for PHP, it was more than 15 years ago. True, for many years I have not used it.
After PHP was Ruby, then it was the HYIP language, many conferences were held, and its framework, Ruby on Rails, became a model for many others. In general, it was in Ruby that many concepts were born, which were later adopted and reborn in other programming languages and their ecosystems.
Then I began to actively develop in JavaScript (node.js). Browsers once competed in speed and security, and in the course of that race, a very productive JavaScript V8 engine was born that was inside the Chromium browser (note: an open browser on which Chrome is based). Some guys came up with taking this engine and screwing it to libevent, which resulted in a wildly productive, asynchronous thing that did other server-side scripting programming languages, albeit with a disgusting syntax and JavaScript ecosystem of that time. I developed it just in this server language.
As for me, so JavaScript was the most terrible language, it was almost impossible to do well, i.e. write compatible code that is convenient to debug, test and develop. But in recent years, JS has greatly improved: the language itself has evolved, many assembly tools have appeared, and most importantly, add-ins that allow us to describe data types, check them for compliance and find potential errors that a developer can make, thereby eliminating most of them. One of these add-ons is TypeScript, for several years I have been using it instead of pure JS.
From Python, of course, there’s no way to go, everything is there, it’s convenient and fast enough, you don’t need to invent anything, you can do almost anything you want. It is well suited for working with data: a lot of mathematical packages, handy tools like Jupyter, take at least the absence of restrictions on the size of the number in mathematical calculations.
My favorite, on which, however, I least work - Elixir. It is based on another functional language Erlang, created by Ericsson for programming telecommunications equipment. Its main feature is parallelism and fault tolerance. By the way, he appeared a long time ago, practically my age. Elixir is just as wildly parallel and fault tolerant. Erlang and Elixir have interesting architectural features, for example, one server can handle several million simultaneous connections by using its own implementation of lightweight processes and exchanging messages through lightweight connections. True, Erlang has an extremely inconvenient and inexpressive syntax.
So, Elixir is, in fact, the same Erlang with all its power, but with a much more pleasant and convenient syntax from Ruby. In Xeteq, we built a productive MTA on it (approx: mail transfer agent, i.e., a mail server). However, I am still afraid to do large-scale projects on Elixir. When using it, you have to write too much yourself: what has been implemented in other languages in the form of ready-made libraries for a long time, you have to do it yourself. Although sometimes it is useful to “reinvent the wheel”, because you can invent something new and worthwhile. And the community is kind of sluggish. All projects have a very small community, for example, let's take the basic web framework plug - it has less than 2 thousand stars, its analog is python flask, which has almost 40 thousand, and ruby sinatra has a little more than 10 thousand.
- Let's talk about the projects. Tell us about your most interesting, significant and important projects, and what is their fate.
- My first serious project was Blonde.ru. In 2009 it was a very ambitious project, the essence of which was that each person, not having special knowledge in contextual advertising, can independently launch an advertising campaign, and simultaneously in all advertising networks and from one simple interface. It was there that I grew very professionally and met people who greatly influenced my life. It so happened that after a few months, the technical director’s chair was vacant, and while they were looking for it, I was appointed by the IO. Some time passed, the chair was empty. The head of the company, Yura Yeliseyev, took and made me a full-fledged technician, I don’t know what he was guided by, I was pretty green for this position, although maybe I saw something in me ( laughs - author’s comment). By the way, it was there that we met with Alexander Shvets (we founded Digital God with him). He came to us to the position of a programmer while still a student of Baumanka and turned out to be the best in this. In general, we then became very good friends. The project is alive, was renamed and transformed several times, entered foreign markets, now it is known as Aori.
- And after Blonde.ru you started doing your first project, right?
- Yes, my friend and I decided to make a commodity widget, in fact, the Yandex.Market advertising network. To make it clearer: there is Yandex.Direct, and there is YAN (Yandex advertising network) - different sites where their blocks are located, and they either catch up with the person through a search, or thematically the show takes place. Yandex.Market didn’t have such a thing, but there was a special content API through which it was possible to receive hot offers from stores and a link where to send those interested. And if the purchase was made, then we were given 50% of the cost that the advertiser paid. Quickly enough, I wrote down the engine - a widget that is placed on the site, it determines the subject, looks what people were interested in, to make an automatic selection of goods from the Market. It worked great. We learned how to arbitrate well (“arbitrage” - buy an advertising space and make it profitable at the expense of another channel), identify good / bad websites, began to actively buy advertising space on many websites and place your widget there. Among the sites was a major automobile. As a result, we had a very fast growth, but, for some strange reason, Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties. began to actively buy advertising space on a variety of sites and place their widget there. Among the sites was a major automobile. As a result, we had a very fast growth, but, for some strange reason, Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties. began to actively buy advertising space on a variety of sites and place their widget there. Among the sites was a major automobile. As a result, we had a very fast growth, but, for some strange reason, Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties. Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties. Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties.
- And yet, gaining all sorts of experience, you continued to invent new projects. What happened next?
- Somehow I set myself the goal of making my own counter to collect data about the behavior of people on websites and calculate statistics based on this - like how Google Analytics and Yandex.Metrica collect their data. The code is put on the site, it keeps track of all visitors' actions on the site, transmits to the server, the server side receives this data and stores it in a distributed database. This project is called Rockstat. Later I was thinking of making a pass-through analytics service on its basis, which, by the way, is now the pain of many companies. But I didn’t have the energy or time to develop, and I didn’t even begin to begin. And it’s good that I didn’t, now I have the opinion that the end-to-end analytics services do their job. They are suitable for those who do not understand analytics and marketing, who do not understand how everything should work, or those who has low needs. In other cases, they run into the lack of integration or incompatibility of the logic of work. But I found an interesting use for raw data collected by the meter.
- What?
- For example, consider multi-channel attribution based on the behavior of the site visitor, and not on the number of the visit. Attribution is a way to distribute the value of the sale between the advertising channels that participated in it. Earlier, the attribution model based on the probabilistic passage of a funnel has become popular, but it is suitable only for sites with a funnel and a not too long sales cycle - these are most online stores. But there is another side: nedvizhka, avtoshka, wedding dresses, where the cycle of decision making is much longer, and there is simply no adequate funnel on the site. For them, I thought up to calculate the index of activity / involvement behavior. At that time, my counter already had several users, this allowed to test the hypothesis, calculate attribution based on the involvement of users in interaction with the site. I took two sites where there is a funnel and not a very short sales cycle. Compared the results of the model based on the probabilistic passage of the funnel and the model based on involvement, they turned out to be significantly correlated. After all, we live in such a time: if the content of the page does not cling or does not give hope, then it will immediately be closed.
I thought that this could be useful and in demand. I went through several companies, told how they can improve the way of evaluating the effectiveness of advertising channels, but it turned out to be too difficult ... It is difficult to understand and adds difficulty in justifying costs. In general, there are very few people who want to do their work really cool, who are driven by what they are doing and who are eager to take new peaks. More often it turns out that, encountering a difficult and incomprehensible decision, they will rather score on it, they will not complicate their lives. He spoke at Data Science Week, Moscow Digital with a story about this model. There were many interested, but no one has matured.
And then there was another turn, some guys came to me to talk about the analyst (I thought so), but it turned out they came to offer me a job, and at the very unusual position of the marketing director for me; and not somewhere there, but to the promising daughter of Sberbank Plazius. This experience deserves a separate story, so perhaps I will miss it.
While still working at Plazius, I began to teach. Together with a friend, together with the Tceh platform, we launched a full-time educational course “Programmatic Advertising Specialist”. The course helped people to increase their competence in the difficult field of high-tech advertising or even get a new profession. And then I ended up at Linkprofit, where I started creating marketing technologies that were primarily focused on the market, and not on meeting the needs of the company. Then the company was in the process of restructuring, without even noticing it, I was appointed head of the new IT company Xeteq, which was included in the LT Digital group of companies.
- Wait, but what about the Rockstat? Abandoned him, since no one understood?
- Once I thought: “Why does Rockstat go dead?” Maybe someone will benefit. Redid the service in the box, added data processing tools and laid out in open source called Alcolytics. And then it started ... A community of very professional guys, whom I used to meet quite rarely, began to gather around the project. They began to implement it at home. I was so amazed! And very competently approached how to use it, I didn’t even have such thoughts. Then I decided for myself that I would only work with enthusiastic and professional, and let the rest go with the forest, I would not even get involved with those who do not know what they want, who should “sell” the idea and then prove its effectiveness.
Almost all my free time I began to spend on the development of this project, I was very motivated by the apparent return. “What for, I used to want to commercialize everything, if I’m preying on the fact that they use it and people like it,” I thought. In general, at first it was somehow scary, suddenly a bug, but I can’t even fix it or they will install it wrong, they will spoil everything, and then they will say that the software does not work (* laughs - author’s comment *). I knew about a dozen servers with Alcolytics, probably there were more of them, but this knowledge was enough for high motivation, I constantly modified the project, and it got better and better.
At one user in Alcolytics data from crm and about calls was added, a through analytics was built. These data were needed to build clever cross-site attribution. This had to be done outside, because There was no way to run the code inside. Then I decided to make it possible to create small scripts to develop my API-NIS, which will allow to take this data into crm or else where, where they should be used. He worked many options, but everywhere there was some flaw. I decided to make a mock service for Docker, but I got so carried away with this process that after two months I realized that I had made my microservice platform, and in fact I wanted to do a simple thing to run scripts. I was very fascinated by the thought: “Why do we need analytics if there is no possibility to automate decision making?” Just look at the numbers a few times, and it is clear in what case what to do. This allows you to do either commercial software, usually with a decent price, or difficult, in terms of implementation and support, open source.
- Yes, simple solutions are not for you, if you do something, then in a big way.
- I imagined how many different automations can be implemented. They will perform various tasks: to manage the advertising budget and rates, or if we talk about RTB advertising, it will be possible to shift users from one segment to another if they change the status in the system. You can do whatever you want. In the process of working on this platform, I invented many “bicycles”, I had never invented so much before. Nobody will praise me for it, but I like them and they turned out very cool. There is another very good argument - it all works on one server for $ 10, keeping the site with 1-2 million visitors per month, without straining at all. Developed frameworks for 2 programming languages that implement all the necessary functionality, organizing interaction with other microservices. By the way
And still, when you make an open source solution, it is not a shame to take a piece from another open source, for example, mobile clients, for the implementation of which it would take a lot of time. In commerce, of course, this is impossible - these are huge reputational risks, and indeed a disgrace. I learned so much cool about open source when I started working with it.
Now this is not a project that solves specific problems, it is a platform that helps to realize your desires, and it is easy to do it, do not bother with complex and boring tech-nar things.
Recently released in alpha testing the 3rd version , returning the original name Rockstat. Documentation is still small, there are many bugs, but users are happy. Among them there are big companies: one airline, and an advertiser from the top 30 of Russia. Of course, there are a lot more users. We have a small community in TG (@rockstats), guys will report on problems and achievements.
- Why and why so much time and personal money you spend on the project, which then spread in open source. What drives you?
- It seems to me that the Rockstat project can greatly change the sphere of digital. It is clear that a standard is suitable for small companies, and giants are building complex solutions, but there are companies with talented specialists and complex needs somewhere in the middle, they are becoming more and more. They create bold and effective solutions with their own hands. The market will follow them or not, it is not known, because we know what Russia is famous for. For some time I will develop the project here, and only he will mature a little, I will try to announce it in other markets.
It was a long way, but now I can honestly say: “I don’t know why I need money, where to put it.” I don’t know why I once thought about money, probably it was imposed by society. Even before open source, I began to teach, I liked to share my knowledge, although at first I thought: “Shouldn’t you keep some knowledge with you?” But no, you need to share with the community, there are many worthy people that your knowledge can be helpful. And the most important thing is that it comes back.
- And what interesting and cool projects did you do as part of your work?
- Oh, I have all such ( laughs - author's note). Now there are several projects in parallel. I will tell, for example, about email service. Mailing lists is already quite an adult direction, there are a lot of players, both local and foreign. However, this niche is not occupied: the players do not have enough opportunities for segmentation, the very high cost of mailings, they are all from 00's, ineffective. Honestly, I did not want to take on this task for a long time. As a result, I decided to start by creating my own mail server - MTA (Message Transfer Agent) - with the minimum necessary functionality. This is still a great job. For 2-3 months of work, we received an MPV (working prototype). In a single user action, multiple servers may start, and a sending system will roll over them. Letters flow like a river - it can scale to huge volumes. Resource consumption is minimal, it is possible to send at least several billion letters a day, and the system will not even notice it. But it’s not so easy to send a billion letters - you need to “warm up” the servers, gradually raising the number of letters sent. I planned to do this project on Elixir, though, personally, I didn’t start implementing, but the guys supported the idea, mastered Elixir and did.
- You have such a great experience, you know a lot and know how, tell me why you need the program “Big Data Specialist” ? What was the need?
- At that time, the largest projects were limited to a cluster of several servers from a relational database. About Hadoop ecosystem and machine learning did not know much. The course gave me useful knowledge, I remember the whole theory, I know where to dig if anything is required; Immediately I can answer, than models on trees differ from linear and gradients. In general, there were so many things there: both the use of Hadoop, and recommender systems, I still scare colleagues with the phrase “collaborative filtering”, and about building a DMP, which was especially important for me - at that time I was working on a similar project. Now we have to work with data a lot: using machine learning to find patterns in the data or to calculate the effectiveness of advertising; determination of microconversion values on sites. At all, The knowledge gained has many uses and I am sure there will be even more. This is already real, without this in any way.
- Let's talk about Digital God - your educational project. How did it come about and what are you teaching?
- This project is out of work, of course, but the need arose just from work: I decided to find programmers in a cunning way (this is a big problem), to make courses for programmers and teach them to Ad- / Mar-Tech. To test the hypothesis, I made a post on Facebook, but programmers did not respond. Digital professionals responded. And it was decided to follow this path: to teach digital programming specialists. It turned out that it is much more interesting and useful because programmers are given marketing and analytical specifics, but they are already doing well, and these guys have suffered: they write TK for a week, then wait for weeks for them to do everything.
My friend (Alexander Shvets) worked the course, launched it, and from the very first it worked - we have a third of the participants survived ( laughs - author's note ). They were very tired and tortured. They had a hard time, they had to sit all weekend long, but they received tremendous professional growth and opportunities. Now the 2nd stream is coming to an end, and the 3rd is approaching the equator. More than half of the people who come to us know that they need to learn programming. They are already experiencing great difficulties due to the lack of possibility of free work with data.
Code is freedom, it is, in essence, a speech for computers. And ready services, applications are restrictions of your opportunities by the interface and the logic developed under the majority. You can do almost anything with code. You need to be thoroughly remarked or think about religion in order to understand what is not decided by programming. And in general, coding is not difficult, because the code is very similar to ordinary English speech. You do not need to be afraid, you have to start learning how to write code, you can’t get anywhere.
Rockstat turned out to be very useful here: we do not start from the basics, but immediately proceed to solving problems that we can face every day. In the second lesson, for example, we solve the problem of classifying visitors to a site according to their behavior, understanding which visitor they wanted, having their history. Dataset is used from the sites of strip clubs (they are also users of Rockstat and have kindly provided a slice of anonymized data). And they found who visited the site for the purpose of employment, and who is interested only in a visit to the club. By the end of the program, everyone should make a course project, according to which we will make a decision on the success of the course. Someone creates his own API, someone considers cross-cutting analytics, one comrade implemented scraper-s for collecting data that cannot be obtained by the API.
Rockstat and Digital God are two major serious projects outside of work. But there is still the most beloved - "Your sweet" . Somehow a friend brought a little caramel in the form of manhood. “Pff, nonsense,” I thought, and threw her in the kitchen, where she lain for about a week. At some point I was bored and I decided to make a candy as large as it really should be. Once in that period I was not working, there was a lot of time, and I was confused. In the meantime, I did, I realized that there was no reason for me to keep this valuable knowledge with me, I had to share it with others. I had a lot of fun doing it, and after a lot of trial and error, a box appeared with all the necessary components and instructions on how to make everything yours.
Graduation 1 of the Digital God stream Digital Rockstar course
- Let's go back to the educational project. Tell me, what skills does a graduate of Digital God have? What can he do?
- The most valuable thing is that the participants received their freedom, they can now describe their tasks in the computer language, and he will take care of the decision, and he will do it on a schedule that he will pledge or even process everything in real time. I think the most interesting conclusions will be in a few months, when graduates will gain more experience.
In addition to digital specialists, we also had owners of small businesses that were involved in digital, because in the modern world everything is very tied to the indicators of the effectiveness of advertising and websites, and it’s very difficult to find an advertising agency that can adequately take on it, and no automation they will not do.
- Why do you not do in principle corporate programs, despite the incoming proposals?
- Knowledge should not be given exclusively to any one big company, besides, according to my observations, the interest of corporate participants is usually lower. Therefore, I don’t want to come to a course for people who are forcibly sent to study. Maybe I'm wrong. If you really need someone, they can take a course with employees of other companies, but, most importantly, they will be in a group of interested people, the atmosphere and impact will be completely different.
- Starting from Blonde.ru, where you quickly grew to a technical director, you always had commands in submission. Please tell us about your approach to work and people in the team.
- Yes, it turns out that I occupy leading positions in most of my professional life, but in my heart I am still a free artist: I need freedom from routine and bureaucracy, this allows me to think freely and find good solutions. I do not like management, for me it is sad, but there is no way to go. In the team, I am not a team leader, now Sasha (Alexander Shvets, a longtime friend and co-founder of DigitalGod) manages the team, and so it’s all better. I am also demanding, I always honestly say in an interview: “We will work a lot, hard and hard. If you are not ready, you don’t have to start. ” There are comrades who believe that they will slip through and quickly run away as a result. But there are those who are ready to get a laptop right at the interview and start working, however, such units.
In general, the great demand for developers and, in general, IT specialists have spoiled people very much: they have learned a little something and have already felt themselves in demand, and, in fact, this sense of false professionalism, which prevents further development. Well, maybe they normally sit in companies where there are a lot of people and processes, and I choose people with whom I have to work, and their motivation is not only financial. And yet I am often mistaken, to be honest, the world is complicated.
- Are there any individual programmers or teams whose work do you follow? What conferences do you attend, which profile blogs / ng channels do you read?
- Frankly speaking, I was thinking about this question for a long time, trying to look at various resources, but I didn’t catch anything. I know how to use a search engine, if I am interested in something concrete, I just take it and look for it.
There are many areas and people that you can be interested in and follow, but I have my own vision and my own course. Perhaps I will be one of those who are being followed. I have a specific marketing and technical perception, with the majority I probably do not follow the path. I do not like banalism, and I do not like to tell the same thing year after year. For many years I have not attended any conferences, because it’s a horror, very few, where there is something interesting or useful, I can go, except to meet with friends.
- How do you have enough to work, own projects and teaching? What is the secret, share.
- The secret is in the absence of personal life ( laughs - author's note ). In general, I do what the soul is for, what I want, I do it. It is hard for me to refuse interesting and cool tasks, and sometimes I take on too much of myself. Now I want to develop my projects, there is a feeling that this can lead to something serious.
- If we talk about life in general, then why do you live?
- I do not have any interesting vision on this topic, to be honest. I perceive life as a computer game - you just pass the levels, choose between different actions that you can calculate, and make a choice. My open source may squeeze out commercial projects from the market, and Digital God courses will set a new level for digital professionals. So it’s interesting that all of this will work out or not. I used to play computer games as a child, then I grew up and realized that life is also a game, but much more exciting, much better graphics, and how many possible lines of development and levels ...
We talked with Dima about programming languages and what solutions he writes on them, about his open source project and why he does it, about the new specialty that they teach, etc.
- Dima, tell us a little about yourself.
- My caramel craft immediately comes to mind, I have long been joking with it, but seriously, I’m a programmer at heart, or an architect, I like to invent and create new solutions, to involve other people in them. For the last 10 years I have been doing marketing technologies, and I got so carried away with them that it turned into a hobby. I developed almost everything: from the advertising management system and DSP to the anti-fraud with hardware monitoring modules. I think I have known Zen, but this is not accurate.
- You have been programming for more than 15 years, tell me, how many programming languages do you know and use in your work and your personal projects?
- I think that counting the number of languages is wrong, the most important skill is to understand several concepts of languages, and then there will be practically no difference on what to write. Yes, of course, each language has its advantages and pitfalls, we should not forget about it.
- And from what languages did you start and what do you use now.
- Now for me there is no particularly big difference on what to write. I started with the development of sites for PHP, it was more than 15 years ago. True, for many years I have not used it.
After PHP was Ruby, then it was the HYIP language, many conferences were held, and its framework, Ruby on Rails, became a model for many others. In general, it was in Ruby that many concepts were born, which were later adopted and reborn in other programming languages and their ecosystems.
Then I began to actively develop in JavaScript (node.js). Browsers once competed in speed and security, and in the course of that race, a very productive JavaScript V8 engine was born that was inside the Chromium browser (note: an open browser on which Chrome is based). Some guys came up with taking this engine and screwing it to libevent, which resulted in a wildly productive, asynchronous thing that did other server-side scripting programming languages, albeit with a disgusting syntax and JavaScript ecosystem of that time. I developed it just in this server language.
As for me, so JavaScript was the most terrible language, it was almost impossible to do well, i.e. write compatible code that is convenient to debug, test and develop. But in recent years, JS has greatly improved: the language itself has evolved, many assembly tools have appeared, and most importantly, add-ins that allow us to describe data types, check them for compliance and find potential errors that a developer can make, thereby eliminating most of them. One of these add-ons is TypeScript, for several years I have been using it instead of pure JS.
From Python, of course, there’s no way to go, everything is there, it’s convenient and fast enough, you don’t need to invent anything, you can do almost anything you want. It is well suited for working with data: a lot of mathematical packages, handy tools like Jupyter, take at least the absence of restrictions on the size of the number in mathematical calculations.
My favorite, on which, however, I least work - Elixir. It is based on another functional language Erlang, created by Ericsson for programming telecommunications equipment. Its main feature is parallelism and fault tolerance. By the way, he appeared a long time ago, practically my age. Elixir is just as wildly parallel and fault tolerant. Erlang and Elixir have interesting architectural features, for example, one server can handle several million simultaneous connections by using its own implementation of lightweight processes and exchanging messages through lightweight connections. True, Erlang has an extremely inconvenient and inexpressive syntax.
So, Elixir is, in fact, the same Erlang with all its power, but with a much more pleasant and convenient syntax from Ruby. In Xeteq, we built a productive MTA on it (approx: mail transfer agent, i.e., a mail server). However, I am still afraid to do large-scale projects on Elixir. When using it, you have to write too much yourself: what has been implemented in other languages in the form of ready-made libraries for a long time, you have to do it yourself. Although sometimes it is useful to “reinvent the wheel”, because you can invent something new and worthwhile. And the community is kind of sluggish. All projects have a very small community, for example, let's take the basic web framework plug - it has less than 2 thousand stars, its analog is python flask, which has almost 40 thousand, and ruby sinatra has a little more than 10 thousand.
- Let's talk about the projects. Tell us about your most interesting, significant and important projects, and what is their fate.
- My first serious project was Blonde.ru. In 2009 it was a very ambitious project, the essence of which was that each person, not having special knowledge in contextual advertising, can independently launch an advertising campaign, and simultaneously in all advertising networks and from one simple interface. It was there that I grew very professionally and met people who greatly influenced my life. It so happened that after a few months, the technical director’s chair was vacant, and while they were looking for it, I was appointed by the IO. Some time passed, the chair was empty. The head of the company, Yura Yeliseyev, took and made me a full-fledged technician, I don’t know what he was guided by, I was pretty green for this position, although maybe I saw something in me ( laughs - author’s comment). By the way, it was there that we met with Alexander Shvets (we founded Digital God with him). He came to us to the position of a programmer while still a student of Baumanka and turned out to be the best in this. In general, we then became very good friends. The project is alive, was renamed and transformed several times, entered foreign markets, now it is known as Aori.
- And after Blonde.ru you started doing your first project, right?
- Yes, my friend and I decided to make a commodity widget, in fact, the Yandex.Market advertising network. To make it clearer: there is Yandex.Direct, and there is YAN (Yandex advertising network) - different sites where their blocks are located, and they either catch up with the person through a search, or thematically the show takes place. Yandex.Market didn’t have such a thing, but there was a special content API through which it was possible to receive hot offers from stores and a link where to send those interested. And if the purchase was made, then we were given 50% of the cost that the advertiser paid. Quickly enough, I wrote down the engine - a widget that is placed on the site, it determines the subject, looks what people were interested in, to make an automatic selection of goods from the Market. It worked great. We learned how to arbitrate well (“arbitrage” - buy an advertising space and make it profitable at the expense of another channel), identify good / bad websites, began to actively buy advertising space on many websites and place your widget there. Among the sites was a major automobile. As a result, we had a very fast growth, but, for some strange reason, Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties. began to actively buy advertising space on a variety of sites and place their widget there. Among the sites was a major automobile. As a result, we had a very fast growth, but, for some strange reason, Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties. began to actively buy advertising space on a variety of sites and place their widget there. Among the sites was a major automobile. As a result, we had a very fast growth, but, for some strange reason, Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties. Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties. Yandex unilaterally broke the contract with us. We tried in every way to find out the reason, but this was never possible, we had to close the project. It was a great sadness, because a lot of energy was invested in this project, and the indicators were very good. Now I’m sure that it was a good experience that taught me how to correctly assess risks. In general, the experience is only positive, and if something does not succeed, it will only help you in the future to avoid even greater difficulties.
- And yet, gaining all sorts of experience, you continued to invent new projects. What happened next?
- Somehow I set myself the goal of making my own counter to collect data about the behavior of people on websites and calculate statistics based on this - like how Google Analytics and Yandex.Metrica collect their data. The code is put on the site, it keeps track of all visitors' actions on the site, transmits to the server, the server side receives this data and stores it in a distributed database. This project is called Rockstat. Later I was thinking of making a pass-through analytics service on its basis, which, by the way, is now the pain of many companies. But I didn’t have the energy or time to develop, and I didn’t even begin to begin. And it’s good that I didn’t, now I have the opinion that the end-to-end analytics services do their job. They are suitable for those who do not understand analytics and marketing, who do not understand how everything should work, or those who has low needs. In other cases, they run into the lack of integration or incompatibility of the logic of work. But I found an interesting use for raw data collected by the meter.
- What?
- For example, consider multi-channel attribution based on the behavior of the site visitor, and not on the number of the visit. Attribution is a way to distribute the value of the sale between the advertising channels that participated in it. Earlier, the attribution model based on the probabilistic passage of a funnel has become popular, but it is suitable only for sites with a funnel and a not too long sales cycle - these are most online stores. But there is another side: nedvizhka, avtoshka, wedding dresses, where the cycle of decision making is much longer, and there is simply no adequate funnel on the site. For them, I thought up to calculate the index of activity / involvement behavior. At that time, my counter already had several users, this allowed to test the hypothesis, calculate attribution based on the involvement of users in interaction with the site. I took two sites where there is a funnel and not a very short sales cycle. Compared the results of the model based on the probabilistic passage of the funnel and the model based on involvement, they turned out to be significantly correlated. After all, we live in such a time: if the content of the page does not cling or does not give hope, then it will immediately be closed.
I thought that this could be useful and in demand. I went through several companies, told how they can improve the way of evaluating the effectiveness of advertising channels, but it turned out to be too difficult ... It is difficult to understand and adds difficulty in justifying costs. In general, there are very few people who want to do their work really cool, who are driven by what they are doing and who are eager to take new peaks. More often it turns out that, encountering a difficult and incomprehensible decision, they will rather score on it, they will not complicate their lives. He spoke at Data Science Week, Moscow Digital with a story about this model. There were many interested, but no one has matured.
And then there was another turn, some guys came to me to talk about the analyst (I thought so), but it turned out they came to offer me a job, and at the very unusual position of the marketing director for me; and not somewhere there, but to the promising daughter of Sberbank Plazius. This experience deserves a separate story, so perhaps I will miss it.
While still working at Plazius, I began to teach. Together with a friend, together with the Tceh platform, we launched a full-time educational course “Programmatic Advertising Specialist”. The course helped people to increase their competence in the difficult field of high-tech advertising or even get a new profession. And then I ended up at Linkprofit, where I started creating marketing technologies that were primarily focused on the market, and not on meeting the needs of the company. Then the company was in the process of restructuring, without even noticing it, I was appointed head of the new IT company Xeteq, which was included in the LT Digital group of companies.
- Wait, but what about the Rockstat? Abandoned him, since no one understood?
- Once I thought: “Why does Rockstat go dead?” Maybe someone will benefit. Redid the service in the box, added data processing tools and laid out in open source called Alcolytics. And then it started ... A community of very professional guys, whom I used to meet quite rarely, began to gather around the project. They began to implement it at home. I was so amazed! And very competently approached how to use it, I didn’t even have such thoughts. Then I decided for myself that I would only work with enthusiastic and professional, and let the rest go with the forest, I would not even get involved with those who do not know what they want, who should “sell” the idea and then prove its effectiveness.
Almost all my free time I began to spend on the development of this project, I was very motivated by the apparent return. “What for, I used to want to commercialize everything, if I’m preying on the fact that they use it and people like it,” I thought. In general, at first it was somehow scary, suddenly a bug, but I can’t even fix it or they will install it wrong, they will spoil everything, and then they will say that the software does not work (* laughs - author’s comment *). I knew about a dozen servers with Alcolytics, probably there were more of them, but this knowledge was enough for high motivation, I constantly modified the project, and it got better and better.
At one user in Alcolytics data from crm and about calls was added, a through analytics was built. These data were needed to build clever cross-site attribution. This had to be done outside, because There was no way to run the code inside. Then I decided to make it possible to create small scripts to develop my API-NIS, which will allow to take this data into crm or else where, where they should be used. He worked many options, but everywhere there was some flaw. I decided to make a mock service for Docker, but I got so carried away with this process that after two months I realized that I had made my microservice platform, and in fact I wanted to do a simple thing to run scripts. I was very fascinated by the thought: “Why do we need analytics if there is no possibility to automate decision making?” Just look at the numbers a few times, and it is clear in what case what to do. This allows you to do either commercial software, usually with a decent price, or difficult, in terms of implementation and support, open source.
- Yes, simple solutions are not for you, if you do something, then in a big way.
- I imagined how many different automations can be implemented. They will perform various tasks: to manage the advertising budget and rates, or if we talk about RTB advertising, it will be possible to shift users from one segment to another if they change the status in the system. You can do whatever you want. In the process of working on this platform, I invented many “bicycles”, I had never invented so much before. Nobody will praise me for it, but I like them and they turned out very cool. There is another very good argument - it all works on one server for $ 10, keeping the site with 1-2 million visitors per month, without straining at all. Developed frameworks for 2 programming languages that implement all the necessary functionality, organizing interaction with other microservices. By the way
And still, when you make an open source solution, it is not a shame to take a piece from another open source, for example, mobile clients, for the implementation of which it would take a lot of time. In commerce, of course, this is impossible - these are huge reputational risks, and indeed a disgrace. I learned so much cool about open source when I started working with it.
Now this is not a project that solves specific problems, it is a platform that helps to realize your desires, and it is easy to do it, do not bother with complex and boring tech-nar things.
Recently released in alpha testing the 3rd version , returning the original name Rockstat. Documentation is still small, there are many bugs, but users are happy. Among them there are big companies: one airline, and an advertiser from the top 30 of Russia. Of course, there are a lot more users. We have a small community in TG (@rockstats), guys will report on problems and achievements.
- Why and why so much time and personal money you spend on the project, which then spread in open source. What drives you?
- It seems to me that the Rockstat project can greatly change the sphere of digital. It is clear that a standard is suitable for small companies, and giants are building complex solutions, but there are companies with talented specialists and complex needs somewhere in the middle, they are becoming more and more. They create bold and effective solutions with their own hands. The market will follow them or not, it is not known, because we know what Russia is famous for. For some time I will develop the project here, and only he will mature a little, I will try to announce it in other markets.
It was a long way, but now I can honestly say: “I don’t know why I need money, where to put it.” I don’t know why I once thought about money, probably it was imposed by society. Even before open source, I began to teach, I liked to share my knowledge, although at first I thought: “Shouldn’t you keep some knowledge with you?” But no, you need to share with the community, there are many worthy people that your knowledge can be helpful. And the most important thing is that it comes back.
- And what interesting and cool projects did you do as part of your work?
- Oh, I have all such ( laughs - author's note). Now there are several projects in parallel. I will tell, for example, about email service. Mailing lists is already quite an adult direction, there are a lot of players, both local and foreign. However, this niche is not occupied: the players do not have enough opportunities for segmentation, the very high cost of mailings, they are all from 00's, ineffective. Honestly, I did not want to take on this task for a long time. As a result, I decided to start by creating my own mail server - MTA (Message Transfer Agent) - with the minimum necessary functionality. This is still a great job. For 2-3 months of work, we received an MPV (working prototype). In a single user action, multiple servers may start, and a sending system will roll over them. Letters flow like a river - it can scale to huge volumes. Resource consumption is minimal, it is possible to send at least several billion letters a day, and the system will not even notice it. But it’s not so easy to send a billion letters - you need to “warm up” the servers, gradually raising the number of letters sent. I planned to do this project on Elixir, though, personally, I didn’t start implementing, but the guys supported the idea, mastered Elixir and did.
- You have such a great experience, you know a lot and know how, tell me why you need the program “Big Data Specialist” ? What was the need?
- At that time, the largest projects were limited to a cluster of several servers from a relational database. About Hadoop ecosystem and machine learning did not know much. The course gave me useful knowledge, I remember the whole theory, I know where to dig if anything is required; Immediately I can answer, than models on trees differ from linear and gradients. In general, there were so many things there: both the use of Hadoop, and recommender systems, I still scare colleagues with the phrase “collaborative filtering”, and about building a DMP, which was especially important for me - at that time I was working on a similar project. Now we have to work with data a lot: using machine learning to find patterns in the data or to calculate the effectiveness of advertising; determination of microconversion values on sites. At all, The knowledge gained has many uses and I am sure there will be even more. This is already real, without this in any way.
- Let's talk about Digital God - your educational project. How did it come about and what are you teaching?
- This project is out of work, of course, but the need arose just from work: I decided to find programmers in a cunning way (this is a big problem), to make courses for programmers and teach them to Ad- / Mar-Tech. To test the hypothesis, I made a post on Facebook, but programmers did not respond. Digital professionals responded. And it was decided to follow this path: to teach digital programming specialists. It turned out that it is much more interesting and useful because programmers are given marketing and analytical specifics, but they are already doing well, and these guys have suffered: they write TK for a week, then wait for weeks for them to do everything.
My friend (Alexander Shvets) worked the course, launched it, and from the very first it worked - we have a third of the participants survived ( laughs - author's note ). They were very tired and tortured. They had a hard time, they had to sit all weekend long, but they received tremendous professional growth and opportunities. Now the 2nd stream is coming to an end, and the 3rd is approaching the equator. More than half of the people who come to us know that they need to learn programming. They are already experiencing great difficulties due to the lack of possibility of free work with data.
Code is freedom, it is, in essence, a speech for computers. And ready services, applications are restrictions of your opportunities by the interface and the logic developed under the majority. You can do almost anything with code. You need to be thoroughly remarked or think about religion in order to understand what is not decided by programming. And in general, coding is not difficult, because the code is very similar to ordinary English speech. You do not need to be afraid, you have to start learning how to write code, you can’t get anywhere.
Rockstat turned out to be very useful here: we do not start from the basics, but immediately proceed to solving problems that we can face every day. In the second lesson, for example, we solve the problem of classifying visitors to a site according to their behavior, understanding which visitor they wanted, having their history. Dataset is used from the sites of strip clubs (they are also users of Rockstat and have kindly provided a slice of anonymized data). And they found who visited the site for the purpose of employment, and who is interested only in a visit to the club. By the end of the program, everyone should make a course project, according to which we will make a decision on the success of the course. Someone creates his own API, someone considers cross-cutting analytics, one comrade implemented scraper-s for collecting data that cannot be obtained by the API.
Rockstat and Digital God are two major serious projects outside of work. But there is still the most beloved - "Your sweet" . Somehow a friend brought a little caramel in the form of manhood. “Pff, nonsense,” I thought, and threw her in the kitchen, where she lain for about a week. At some point I was bored and I decided to make a candy as large as it really should be. Once in that period I was not working, there was a lot of time, and I was confused. In the meantime, I did, I realized that there was no reason for me to keep this valuable knowledge with me, I had to share it with others. I had a lot of fun doing it, and after a lot of trial and error, a box appeared with all the necessary components and instructions on how to make everything yours.
Graduation 1 of the Digital God stream Digital Rockstar course
- Let's go back to the educational project. Tell me, what skills does a graduate of Digital God have? What can he do?
- The most valuable thing is that the participants received their freedom, they can now describe their tasks in the computer language, and he will take care of the decision, and he will do it on a schedule that he will pledge or even process everything in real time. I think the most interesting conclusions will be in a few months, when graduates will gain more experience.
In addition to digital specialists, we also had owners of small businesses that were involved in digital, because in the modern world everything is very tied to the indicators of the effectiveness of advertising and websites, and it’s very difficult to find an advertising agency that can adequately take on it, and no automation they will not do.
- Why do you not do in principle corporate programs, despite the incoming proposals?
- Knowledge should not be given exclusively to any one big company, besides, according to my observations, the interest of corporate participants is usually lower. Therefore, I don’t want to come to a course for people who are forcibly sent to study. Maybe I'm wrong. If you really need someone, they can take a course with employees of other companies, but, most importantly, they will be in a group of interested people, the atmosphere and impact will be completely different.
- Starting from Blonde.ru, where you quickly grew to a technical director, you always had commands in submission. Please tell us about your approach to work and people in the team.
- Yes, it turns out that I occupy leading positions in most of my professional life, but in my heart I am still a free artist: I need freedom from routine and bureaucracy, this allows me to think freely and find good solutions. I do not like management, for me it is sad, but there is no way to go. In the team, I am not a team leader, now Sasha (Alexander Shvets, a longtime friend and co-founder of DigitalGod) manages the team, and so it’s all better. I am also demanding, I always honestly say in an interview: “We will work a lot, hard and hard. If you are not ready, you don’t have to start. ” There are comrades who believe that they will slip through and quickly run away as a result. But there are those who are ready to get a laptop right at the interview and start working, however, such units.
In general, the great demand for developers and, in general, IT specialists have spoiled people very much: they have learned a little something and have already felt themselves in demand, and, in fact, this sense of false professionalism, which prevents further development. Well, maybe they normally sit in companies where there are a lot of people and processes, and I choose people with whom I have to work, and their motivation is not only financial. And yet I am often mistaken, to be honest, the world is complicated.
- Are there any individual programmers or teams whose work do you follow? What conferences do you attend, which profile blogs / ng channels do you read?
- Frankly speaking, I was thinking about this question for a long time, trying to look at various resources, but I didn’t catch anything. I know how to use a search engine, if I am interested in something concrete, I just take it and look for it.
There are many areas and people that you can be interested in and follow, but I have my own vision and my own course. Perhaps I will be one of those who are being followed. I have a specific marketing and technical perception, with the majority I probably do not follow the path. I do not like banalism, and I do not like to tell the same thing year after year. For many years I have not attended any conferences, because it’s a horror, very few, where there is something interesting or useful, I can go, except to meet with friends.
- How do you have enough to work, own projects and teaching? What is the secret, share.
- The secret is in the absence of personal life ( laughs - author's note ). In general, I do what the soul is for, what I want, I do it. It is hard for me to refuse interesting and cool tasks, and sometimes I take on too much of myself. Now I want to develop my projects, there is a feeling that this can lead to something serious.
- If we talk about life in general, then why do you live?
- I do not have any interesting vision on this topic, to be honest. I perceive life as a computer game - you just pass the levels, choose between different actions that you can calculate, and make a choice. My open source may squeeze out commercial projects from the market, and Digital God courses will set a new level for digital professionals. So it’s interesting that all of this will work out or not. I used to play computer games as a child, then I grew up and realized that life is also a game, but much more exciting, much better graphics, and how many possible lines of development and levels ...