Overview of the best reports with HighLoad ++ 2017
In the next few articles I will talk about the best ( according to the participants ) reports of HighLoad ++ 2017. The organizers have kindly opened access to the videos that you can right here and watch.
For me, this is the opening of the year - at a powerful technology conference, the first place is occupied by a report, although from a techie, but about MANAGEMENT. Of course, one can argue that humanities are more willing to give ratings and, by default, a more loyal audience, but the fact remains.
A story about the features and problems in the transition from an engineer to a leader. The report can be divided into two parts. The first, which reveals the topic of the report, is about problems when moving from an engineering staff to a leading one, and the second is advice to leaders on how to grow and care for new leaders. In fact, the issue under consideration covers all levels of the hierarchical ladder, because “Sores” are characteristic for moving to any next level of control.
It is always interesting to listen to a professional who honestly and openly shares his experience, both successful and unsuccessful.
The first and very cool tip is “be honest about what you do well and what you can do.” Indeed, this is essentially the only support that you will have when moving to a new level. In fact, Artyom recommends conducting a self-analysis session and honestly (this is very important) understand where your weaknesses are, so that under no circumstances should you use them, or at least not use them in the first place. If you have a good memory, you are a good communicator - use these qualities.
If you still managed to become a leader, then you must be prepared for the fact that the following unpleasant moments will surely meet on your way:
The second part of the story is devoted to the issues of cultivating and searching for leaders within the unit: search, competency testing, recruitment, and most importantly support at the first stages of the formation of a new leader. You can’t leave a person no matter how cool he seems.
In conclusion, I’ll bring my hit parade of episodes from this story:
Artyom gives an example of how he and his ward tried to create a leader, but in the end they recognized that it was all wrong and the engineer remained an engineer. At the same time, he emphasizes that there should be a legend in case of failure, when the team will have to explain why a person returned to his previous position (my personal experience suggests that this is the most difficult task in such a situation).
In second place is the episode when, to the surprise of part of the audience, the phrase suddenly sounds - Being an engineer is much funnier in fact, and switching to managers is not a transition to the country of pink ponies.
Well, the story closes that after 10! years, nothing human is alien. Proof.
It remains to note that 25 minutes of questions and answers after the end of the report once again demonstrated the relevance of the topic of management and career growth in IT.
In second place, perhaps, is one of the most charismatic people in IT today. Super-professional, author of the popular search engine Sphinx, “Voronezh cattle” (don’t think, this is a quote from a report at one of the previous conferences :)), the merry fellow and the swindler with a “left-handed” point of view on many questions is Andrei Aksyonov. Those who have not heard his reports at least once vehemently recommend it.
Recently, Andrew began to talk about general technical topics. In this report, the focus on compression is an overview of compression methods and approaches for different types of data. We are talking about general approaches and algorithms that are applicable in almost all areas of programming regardless of what language you write in C ++ or, excuse me, in PHP.
In general, compression may not be necessary to solve your specific problem, but nevertheless, at any moment a situation may arise when compression will save you. Even if you don't store anything (yes, compressing large array variables is sometimes very useful as well). I want to give a simple household example on my own - at the moment when a disaster occurred with disk space, transcoding a video with a lower bitrate came to the rescue, which in essence is one example of compression, albeit with a slight loss in quality, as well as image compression, which does not give a good result, but on a large number of files - the exhaust is very noticeable.
The whole story is filled with excellent examples, in fact, everything is built precisely on the analysis of examples of solving certain problems. IMHO is one of the best formats for presenting information.
The story has two parts:
So, about coding (not to be confused with coding).
“Frequent packing, rare inflating.” All. We diverge. Inflation during compression sounds seditious, but in fact it’s not so.
“Intuition should not be trusted when it comes to large amounts of information. It is necessary to check. Build frequency tables. In tests, most likely the most common character is a space and a carriage return. And far from always remember about it.
But everything is not always great. An excellent example is given about how through unsuccessful compression you can get a shot in the leg. The frequent (space) was encoded into one bit, and the rare inflated to 9 bits. As a result, we got 3% savings, but at the same time we deprived ourselves of the possibility of quick access to the elements of the text array, because the elements are now unclear how long. It turns out that the principle as a whole is true, but somehow it was not very applied.
The conversation goes on about how the coding algorithm evolves step by step - “we are looking for a balance”, where is it really frequent and where is rare. As a result, we got an algorithm that compresses a little worse, BUT, it eliminates the work of bit reading, which, all other things being equal, is more likely a gain.
“Do not be shy to inflate the rare very much.” True, the same nuance pops up again - it does not always work. It all depends on the data set. In general, compression is all about analyzing data sets, selecting algorithms and methods. With a good choice, it is your result that can surpass all classical algorithms, both in compression quality and in speed.
What did the beats do after all? In fact, nothing - it is simply more difficult to implement the read operation, which leads to a decrease in speed, albeit to an increase in space saving. Then bytes? BUT, as Andrew rightly observes, if you compress bytes into bytes, you may be surprised to find that the compression ratio is equal to one. In other words, the statement is confirmed once again that everything has its time and place.
By the way, it is worth paying attention that almost all the examples that Andrei cites can to some extent be easily reproduced in most modern programming languages with minimal refinement, and most importantly, they are simple to understand.
Everything is much wider here, because Each type of content has its own characteristics. Consistently over the years, we look at the evolution and birth of algorithms:
Alas, since then they haven’t come up with anything fundamentally new. Inside all the transformations, nothing outstanding, and most importantly almost everything uses the same LZ, but in its more modern implementation LZ77. Proof.
But what if it's not text? For example, in the case of search engines, when almost all the data is huge arrays of numbers. And then mankind came up with a solution - delta coding and various variations on its theme. But here, Andrei reminds - not everything is so cool, there will always be situations when this does not work. And again we return to the idea of selecting an algorithm for data and tasks. For those who want to expand their horizons, here is a list of keywords (not all) that were sounded in the report.
In general, if we summarize the report, we can draw the following conclusions:
In the meantime, we invite you to continue discussing the topic of team lead at our TeamLead Conf conference , and compression and generally programming with the help of correctly growing hands at the Backend Conf server programming conference .
Goth2Boss: breaking and waste during the transition from engineer to team lead / Artyom Kalichkin
For me, this is the opening of the year - at a powerful technology conference, the first place is occupied by a report, although from a techie, but about MANAGEMENT. Of course, one can argue that humanities are more willing to give ratings and, by default, a more loyal audience, but the fact remains.
A story about the features and problems in the transition from an engineer to a leader. The report can be divided into two parts. The first, which reveals the topic of the report, is about problems when moving from an engineering staff to a leading one, and the second is advice to leaders on how to grow and care for new leaders. In fact, the issue under consideration covers all levels of the hierarchical ladder, because “Sores” are characteristic for moving to any next level of control.
It is always interesting to listen to a professional who honestly and openly shares his experience, both successful and unsuccessful.
The first and very cool tip is “be honest about what you do well and what you can do.” Indeed, this is essentially the only support that you will have when moving to a new level. In fact, Artyom recommends conducting a self-analysis session and honestly (this is very important) understand where your weaknesses are, so that under no circumstances should you use them, or at least not use them in the first place. If you have a good memory, you are a good communicator - use these qualities.
If you still managed to become a leader, then you must be prepared for the fact that the following unpleasant moments will surely meet on your way:
- A burning desire to protect their employees from external threats (Good Corporation has a special term, Shit Umbrella, which very clearly illustrates this behavioral pattern). From the point of view of Artyom, this is one of the greatest evils for the team and for the leader, because the unit becomes a black box for the outside world, problems are masked, the level of criticality towards oneself and one's mistakes is reduced. All this leads to a decrease in quality and interest in work. Any situation that takes a person out of the comfort zone encourages him to connect additional resources and work as efficiently and productively as possible. If you want to get it - close the umbrella, although not immediately. By the way, about how to close the umbrella correctly, there is a question and answer after the main report. As an advice and illustration, Artyom cites an episode from the life of his colleague, who, in the case of the fakap, came to the team and said, “We screwed up.” The important thing here is that there is no one specific who is guilty. As a result, the team rallies, mobilizes and corrects what caused this arrival.
- Difficulty in perceiving delayed results. Here we are talking about the fact that the engineer has quite a concrete result of work that can be achieved in the foreseeable future and realized. As for the leader, then everything is more complicated, because the very essence of leadership does not imply an objective assessment of the actions of the leader. It is unlikely that anyone can say unequivocally - "yes, we have implemented processes successfully." More often you can hear the epithets “in general”, “enough”, “almost”, etc. Those. the final result can be achieved and will be achieved, but not immediately, and most importantly, all the time, while there is no result, the head is haunted by the feeling that nothing is happening and some uncertainty that what is being done is being done towards the goal.
- The pursuit of ideal solutions. It is important to remember that “the best enemy of the good”. Unfortunately, the current realities are such that the manager is forced in almost 100% of cases to make a decision in the conditions of lack of information, time for analysis or lack of resources to develop the very ideal solution. It sounds a little strange, but Artyom claims that if you made the perfect decision, then you must have missed something somewhere. And at least spent too much time on its development. I’ll add from myself that such decisions are made quite often, but it’s rather luck and a lot of experience.
- A relevant slide is the
Beach of all newly minted leaders - the desire to do everything yourself and generally do something with your hands. A very important point - it is important to distinguish the problem of delegation from the desire to work with hands. Not always the desire to work with hands is associated with a lack of the ability to delegate. Most just like to do engineering work and slightly reduce the very “professional breaking”. Extremely useful advice - work with your hands, but shift the direction of your activity from tactical to strategic tasks. Quote: “Former engineers do not exist.”
The second part of the story is devoted to the issues of cultivating and searching for leaders within the unit: search, competency testing, recruitment, and most importantly support at the first stages of the formation of a new leader. You can’t leave a person no matter how cool he seems.
In conclusion, I’ll bring my hit parade of episodes from this story:
1st place.
Artyom gives an example of how he and his ward tried to create a leader, but in the end they recognized that it was all wrong and the engineer remained an engineer. At the same time, he emphasizes that there should be a legend in case of failure, when the team will have to explain why a person returned to his previous position (my personal experience suggests that this is the most difficult task in such a situation).
2nd place.
In second place is the episode when, to the surprise of part of the audience, the phrase suddenly sounds - Being an engineer is much funnier in fact, and switching to managers is not a transition to the country of pink ponies.
3rd place.
Well, the story closes that after 10! years, nothing human is alien. Proof.
It remains to note that 25 minutes of questions and answers after the end of the report once again demonstrated the relevance of the topic of management and career growth in IT.
I want to squeeze everything / Andrey Aksyonov
In second place, perhaps, is one of the most charismatic people in IT today. Super-professional, author of the popular search engine Sphinx, “Voronezh cattle” (don’t think, this is a quote from a report at one of the previous conferences :)), the merry fellow and the swindler with a “left-handed” point of view on many questions is Andrei Aksyonov. Those who have not heard his reports at least once vehemently recommend it.
Recently, Andrew began to talk about general technical topics. In this report, the focus on compression is an overview of compression methods and approaches for different types of data. We are talking about general approaches and algorithms that are applicable in almost all areas of programming regardless of what language you write in C ++ or, excuse me, in PHP.
In general, compression may not be necessary to solve your specific problem, but nevertheless, at any moment a situation may arise when compression will save you. Even if you don't store anything (yes, compressing large array variables is sometimes very useful as well). I want to give a simple household example on my own - at the moment when a disaster occurred with disk space, transcoding a video with a lower bitrate came to the rescue, which in essence is one example of compression, albeit with a slight loss in quality, as well as image compression, which does not give a good result, but on a large number of files - the exhaust is very noticeable.
The whole story is filled with excellent examples, in fact, everything is built precisely on the analysis of examples of solving certain problems. IMHO is one of the best formats for presenting information.
The story has two parts:
- Coding with code examples and a visual demonstration of the pros and cons of different algorithms.
- Conversion, overview, because the variability is too high - we transform the images in one way, the video in another. But there are examples too, and they are close to many.
So, about coding (not to be confused with coding).
Tip 1.
“Frequent packing, rare inflating.” All. We diverge. Inflation during compression sounds seditious, but in fact it’s not so.
Tip 2.
“Intuition should not be trusted when it comes to large amounts of information. It is necessary to check. Build frequency tables. In tests, most likely the most common character is a space and a carriage return. And far from always remember about it.
But everything is not always great. An excellent example is given about how through unsuccessful compression you can get a shot in the leg. The frequent (space) was encoded into one bit, and the rare inflated to 9 bits. As a result, we got 3% savings, but at the same time we deprived ourselves of the possibility of quick access to the elements of the text array, because the elements are now unclear how long. It turns out that the principle as a whole is true, but somehow it was not very applied.
The conversation goes on about how the coding algorithm evolves step by step - “we are looking for a balance”, where is it really frequent and where is rare. As a result, we got an algorithm that compresses a little worse, BUT, it eliminates the work of bit reading, which, all other things being equal, is more likely a gain.
Tip 3.
“Do not be shy to inflate the rare very much.” True, the same nuance pops up again - it does not always work. It all depends on the data set. In general, compression is all about analyzing data sets, selecting algorithms and methods. With a good choice, it is your result that can surpass all classical algorithms, both in compression quality and in speed.
What did the beats do after all? In fact, nothing - it is simply more difficult to implement the read operation, which leads to a decrease in speed, albeit to an increase in space saving. Then bytes? BUT, as Andrew rightly observes, if you compress bytes into bytes, you may be surprised to find that the compression ratio is equal to one. In other words, the statement is confirmed once again that everything has its time and place.
By the way, it is worth paying attention that almost all the examples that Andrei cites can to some extent be easily reproduced in most modern programming languages with minimal refinement, and most importantly, they are simple to understand.
Now about the conversion.
Everything is much wider here, because Each type of content has its own characteristics. Consistently over the years, we look at the evolution and birth of algorithms:
- 70s - Lempel-Ziv (the same LZW, https://ru.wikipedia.org/wiki/Lempel__Ziva_—_Welcome algorithm, which is still used in specifications, for example, in pdf and gif)
- 80s - PPM https://en.wikipedia.org/wiki/%D0%90%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC_%D1% 81% D0% B6% D0% B0% D1% 82% D0% B8% D1% 8F_PPM (7zip, RAR, WinZip and a few more)
- The 90s and 2000s were held under the motto of productivity growth and additional tuning of what was invented earlier.
- 2010s - Brotli - https://ru.wikipedia.org/wiki/Brotli (mainly used by browsers)
Alas, since then they haven’t come up with anything fundamentally new. Inside all the transformations, nothing outstanding, and most importantly almost everything uses the same LZ, but in its more modern implementation LZ77. Proof.
But what if it's not text? For example, in the case of search engines, when almost all the data is huge arrays of numbers. And then mankind came up with a solution - delta coding and various variations on its theme. But here, Andrei reminds - not everything is so cool, there will always be situations when this does not work. And again we return to the idea of selecting an algorithm for data and tasks. For those who want to expand their horizons, here is a list of keywords (not all) that were sounded in the report.
In general, if we summarize the report, we can draw the following conclusions:
- The approaches are all quite simple, the main thing is to think which best compression and conversion techniques make sense.
- For 50 years they did not come up with anything new, although they tried diligently. Everything new is a variation on the theme of the old. Therefore, most likely, one of the above will be suitable for your task, especially if you carefully follow the advice. And if you don’t believe it, look and listen to how cool search engines and databases work. Under the hood, everything is what the report said.
- Do not be shy to drive benchmarks and compare results. According to the results, you can easily abandon gzip in favor of zstd.
- Aksyonov is very cool, although it will seem to many people that he’s slandering too much, but he is like that. I assure you, all his stories are worthy of attention.
What will happen next?
3. Fraud in mobile applications: how to define and detect / Vadim Antonyuk ;
4. We make our firmware for the IP camera on Rust / Max Lapshin ;
5. BigMail: how we built DataLake in the Russian Post / Alexey Vovchenko;
6. Cheaper, more reliable, easier. Storage of petabytes of video and photo in OK / Alexander Khristoforov;
9. Java and Linux - operating features / Alexey Ragozin;
10. Details on how Causal Consistency is implemented in MongoDB / Mikhail Tyulenev (Misha Tyulenev) ;
In the meantime, we invite you to continue discussing the topic of team lead at our TeamLead Conf conference , and compression and generally programming with the help of correctly growing hands at the Backend Conf server programming conference .