Additional lectures of the course "Design of high-loaded systems" (autumn 2018) in Technopolis
We continue to publish lectures on the course “Design of High-Load Systems”, which is read to students of St. Petersburg Polytechnic University by a team of engineers from Odnoklassniki as part of the two-year program “Java Developer of High-Load Applications” of the Technopolis project (a joint Mail.Ru Group project and SPbPU). In 2017, 10 lectures were read and posted (30 hours of video)But the theme of Highload is so extensive that it is impossible to cover everything in one semester. We only briefly plunged into the main aspects of Highload-development, each of which is worth a separate course. This year we continue to close white spots and bring to your attention a set of six lectures on new topics: we start with parallel computing and livecoding the first stage of the student course project, after which we dive into the JVM monitoring and diagnostic tools, and then proceed to resiliency problems. And after the lecture on advanced algorithms that are relevant in high-load projects, we end the cycle with a lecture on existing approaches to replication and their applicability to various tasks.
The first ten lectures .
List of new lectures:
- Actor Model. Future. Reactive Streams (Vadim Tsesko incubos )
- Livecoding second stage of the project (Vadim Tsesko incubos )
- Monitoring and Diagnostics JVM (Andrey Pangin apangin )
- Site Reliability Engineering (Anton Ivanov keyplayer )
- "Modern" data structures (Dmitry Schitinin dormidoncheg )
- Replication (Dmitry Schitinin dormidoncheg )
Lecture 11. Actor Model. Future. Reactive Streams (Vadim Tsesko incubos )
Presentation
Video on Technostrim channel
The lecture discusses modern approaches to the organization of parallel computing, which are gaining popularity recently: Actor Model based on message passing in the Akka implementation; abstract Future / Promise, combinators and callbacks for the organization of asynchronous data processing during IO; and also reveals the essence of Reactive Streams for streaming data processing with back pressure. There are numerous examples of using approaches in existing systems, as well as links for self-study.
Lecture 12. Livecoding second phase of the project (Vadim Tsesko incubos )
Course project
Video on the channel
Tehnostrim Livecoding the basic solution of the project, which students of Technopolis must perform this year. Similar to the first lesson, with the only fundamental difference: this time we switched to a high-performance web server from one-nio. Also this time we recorded a video with a more detailed demonstration of the implementation of components.
Lecture 13. Monitoring and diagnostics of JVM (Andrey Pangin apangin )
The presentation of the
video on the channel Tehnostrim
Lecture introduces the means to monitor a Java-based application and analyze typical problems: deadlock, memory leaks, performance bottlenecks. Both standard utilities from the JDK and popular third-party software are considered, and Java-profilers are considered. It shows how JMX, JVM TI and other interfaces help in developing your own monitoring and diagnostic tools.
Lecture 14. Site Reliability Engineering (Anton Ivanov keyplayer )
Video presentation on the channel Tehnostrim
Lecture is devoted to the basics of writing fault-tolerant code. The typical errors are analyzed, which in practice lead to the fall of highly loaded systems. The first part deals with the problems that arise at the service level: resource pools, their overload, slow start and graceful shutdown. In the second part, we turn to the level of interservice interaction, we consider in detail the types of timeouts, balancing techniques and problems of the rerays. In the final part, issues of monitoring and reactions to incidents are addressed.
Lecture 15. "Modern" data structures (Dmitry Schitinin dormidoncheg )
Presentation
Video on the channel Tehnostrim
Lecture is devoted to data structures, which, as a rule, are not considered in classical courses, but are effectively used in practice in highly loaded distributed systems. Often these are probabilistic data structures, compact and fast, but giving an approximate result. For example, using HyperLogLog, you can estimate the number of different elements in a billion-item dataset, at a cost of no more than a kilobyte, and the result will differ from the exact one by only a few percent. A separate section is devoted to approaches to the calculation of quantiles, their strengths and weaknesses are considered.
Lecture 16. Replication (Dmitry Shchitinin dormidoncheg )
The presentation of the
video on the channel Tehnostrim
Replication, along with shardirovanie underlie highly loaded fault-tolerant distributed systems. Replication with one dedicated leader is widely used in classic RDBMS. A configuration with several selected leaders can occur with systems operating in several DCs, and also arises in problems of collaborative editing. Replication without a dedicated leader lies at the heart of NoSQL databases such as Cassandra, Voldemort, Riak, and others. The lecture covers each approach, scope, strengths and weaknesses, and key implementation details.
The playlist of all lectures is on the link , as well as on Youtube .
Course project is here .
Various solutions can be viewed among the closed pull requests .
Broadcasts and video recordings of other courses of the Technopolis project can be found in the official project group in OK .
We also remind you that on Tehnostrim channel there are current lectures and master classes on programming and data analysis from IT-specialists from all educational projects of Mail.Ru Group - Technoatom, Technopark, Technopolis, Technosphere and Technotrak.