The best reports of the JPoint 2018: Java / JVM and its performance, Kotlin, Spring, Docker
We have already posted videos of JPoint 2018 reports on YouTube and, specifically for the Java hub on Habré, we made a traditional selection of the best of them according to conference visitors.
As usual, the top “junior” reports, in the end - with the highest rating. Of course, this does not mean that one report is much worse than another: if you change the calculation method, places can easily change. In reality, we changed it, now the “soft quorum” version of the rating is used, taking into account the number of participants present at the report. This approach has its drawbacks (for example, more people come to the keyout than to the regular report, simply because the audience has no choice), but in general it gives a better picture of what happened.
Under the cut - and videos of the best reports, and links to their presentations, and short descriptions, and a link to the full playlist.
The full playlist with all the videos that will be discussed below is available here .
10. Linux container performance tools for JVM applications
The container revolution has overwhelmed everyone - the world of Java has not been without it. Nevertheless, Java has always had some problems and peculiarities with it, which will be discussed in the report of Sasha Goldstein.
Sasha is a serial creator of performance hardcore. Last year at JPoint, he gave a great talk about using the Berkeley Packet Filter for JVM (I desperately recommend watching the recording on YouTube), and it was only a matter of time before he got to the detailed containerization parsing. The world goes to the clouds and dockers, which in turn brings us many new problems. As you can see, most low-level debugging and profiling systems after their application to containers are overgrown with various features and jambs. Sasha skated around the main scenarios (CPU load, IO responsiveness, access to shared databases, etc.) through the prism of using modern tools on the GNU / Linux platform, including BCC and perf.
This specificity is needed not by all. This is one of the reasons why the report was only in tenth place - it was attended by about two hundred people (compare with more than a thousand from Tolkachev and Borisov), and our rating calculation algorithm is very sensitive to the size of the audience.
For those who are somehow connected with containers, this report is a must-see. Save a lot of time on an independent search for all the details using the method of complete enumeration of the countless number of utilities and ways to use them.
I would also like to note that Sasha’s report from all the top 10 is almost the densest in terms of the amount of practical information per unit of time. Almost from each slide you can copy yourself some useful things. In order not to engage in a detailed retelling, I will show only the surface structure of the presentation:
- How are containers arranged?
- Control groups: CPU, memory, block I/O;
- Namespaces: pid namespace, mount namespace, network namespace;
- Разница между возникающими проблемами:
- На хосте;
- В контейнере;
- Примеры проблем с решениями:
- Подключение к JVM?
- Данные о перформансе JVM?
- Демо JVM-инструментов на хосте;
- Инструменты для получения информации о ресурсах:
- sidecar для мониторинга;
- docker stats;
- htop + cgroup ID;
- nsenter или docker exec;
- Демо мониторинга ресурсов контейнера;
- Профилирование CPU-контейнеров:
- perf на хосте: -G, symbol mapping, PID mapping, sharing perf map (perf-map-agent);
- Демка с флеймграфами; оос;
- perf в контейнере в непривилегированном режиме;
- honest-profiler, async-profiler, perf vs async-profiler;
- Демка async-profiler;
- BCC — инструмент для профилирования (само решает рядо проблем, но работает только на Linux 4.9+);
- Другие способы мониторинга:
- cAdvisor, SysDig, New Relic, DataDog…
9. Once a year, gardens bloom: debriefing exactly-once semantics by Apache Kafka
Victor Gamov has a completely unfair advantage over the other speakers. This is the man who released nearly two hundred issues of the podcast Debriefing, the author of a heap of reports, texts, posts and even the book “Enterprise Web Development” of O'Reilly. The very presence of Gamow makes anything better. I’m sure he could, with the same intonation, tell not about Kafka, but about the growth of geranium on the windowsill, and it would still be fascinating.
Here we have a special case: initially holivornaya topic about the semantics of "exactly once". Kafka was rushed to use everything on an extraordinary scale, which required reworking the traditional semantics of message delivery. Kafka didn’t put a finger in his mouth - they changed the correct protocol and message format on the go and did everything they needed. Throughout the report, Victor tells how it works inside and what it affects.
Two news, good and bad. Good: everything is set up in Kafka. The bad: in Kafka everything is set up. To live with it, you need to understand how it works and where to climb with your dirty hands.
You could expect some kind of smoothie from Victor, but then you got a desperate hardcore, and something like this:
- Введение в Кафку:
- Логи и оффсеты
- Реплики и лидеры
- Клиенты: продюсеры и консюмеры
- Как работает на примерах
- Протокол продюсера
- Как работает на примерах
- Протокол консюмера
- Модель обработки
- Прочитали, посчитали, записали
- Виды сбоев (все сломалось, зомби)
- Семантика обработки
- At-least-once, at-most-once, exactly-once
- Cемантика «exactly once»
- С примерами
- небезопасные повторы
- неатомарная запись со смещением
- Чиним первую слабость: идемпотентный продюсер
- Чиним вторую слабость:
- Снэпшоты Чэнди-Лэмпорта
- Как жить со сбоями?
- Hard: сбой во время обработки
- Soft: сбой до или во время снятия снапшота
- Транзакции в Кафке
- Два типа маркеров (COMMIT, ABORT)
- Атомарная запись на множество партиций (Включая !_consumer_offsets)
- Снэпшоты Чэнди-Лэмпорта
- Чиним третью слабость: zombie fencing
- Изоляция чтения у консюмера
- end-to-end eos
- Kafka Connect Source
- Kafka Streams
- Kafka Connect Sink
8. Fight Russian Hackers with Kafka Streams and Firehose API
Again Kafka! Again Gamov! Yes, and with Baruch Sadogursky ( jbaruch ) now. Nevertheless, this report is not a smoothie, but a very concrete practical thing about how, on the Bintray platform (of which Baruch is the god), Apache Kafka (for which Victor is responsible) and Firehose analyze behavioral patterns, process large amounts of data.
If the previous report about Kafka's insides had 245 slides, then this one is only 17. That's because you need to watch it! This is primarily a lively dialogue between leading and live demos. Do not linger, open vidosik and look.
7. Hardware Transactional Memory in Java
Nikita Koval ( ndkoval ) is a researcher at JetBrains in the Kotlin team and a PhD student at IST Austria (at the time of the report he was at Devexperts). His report contrasts strongly with the "Russian hackers", because it is not a light entertaining read, but a story about the intricate insides of a VM. If you look at the slides , you can find 150 sheets, most of which are code.
If you were at JBreak in the beginning of 2018, you could find Nikita’s story about completely different things - writing a fast multi-threaded hash table using the power of modern multi-core architectures and special algorithms. Agree, Nikita has a style . I immediately recall Shipilev.
This time we will talk about transactional memory, which gradually appears in modern processors, but which is not yet clear how to use an ordinary person from the JVM world. Outside the JVM, of course, everything is easier. But imagine how you tell the usual Spring web developer: "Yes, you just edit the vmstructs, load your intrinsics, rebuild OpenJDK and you're done!" And he said: “Well, of course, I do it every day!”. Nikita just very clearly talks about how to use, what optimizations are already in OpenJDK and how to perform transactions directly from Java code.
- Введение: почему нам нужна многопоточность
- Подходы к построению алгоритмов
- Грубая блокировка
- Тонкая блокировка
- Неблокирующая синхронизация
- Все три вида иллюстрируются на задаче-примере про игрушечный банк
- Многопоточность — это сложно. Что делать?
- Транзакции в идеальном мире. Просто пиши atomic!
- Откуда взять atomic:
- Software Transactional Memory (STM). Scala STM, NOrec, корутины.
- Hardware Transactional Memory (HTM). Haswell, Power 8.
- Hybrid Transactional Memory
- Intel RTM на примерах
- XBEGIN, XEND, XABORT, XTEST
- Intel RTM + Java
- Lock elision
- Intrinsics: interpreter, C1, C2
- Coarse-grained / Lock-free + RTMSupport на графиках
6. We profile up to microseconds and processor instructions
Sergey Melnikov ( RainM ) from Raiffeisenbank brought us the second profiling report. Interestingly, before taking up the low-latency code in Java, he worked at Intel as a performance engineer of compilers for C / C ++ / FORTRAN languages. This report also has a perf! :-) There is also about the hardware features of the processors and Intel Processor Trace technology, which allows you to take the next step in profiling accuracy and to reconstruct the execution of the program section. There are quite a few such reports (for example, you can find the Andi Kleen report at the Tracing Summit 2015), they usually leave a lot of questions and aren’t very practical in Java. Here we not only have a person who has visited both worlds (both Intel and Java in the bank), but also who is able to clearly explain complex topics.
- Что это и зачем нужно
- Предметная область — low latency приложения
- Пример Московской биржи
- Выбираем профилировщик
- Как профилировать? Сэмплирующие и инструментирующие профилировщики
- Учим профилировщик собирать подробный профиль
- Как запускать perf
- Как смотреть в профиль, в call stack
- perf-map-agent, sysctl, dmesg…
- Используем события PMU/PEBS в perf
- Intel Processor Trace – что это такое и как профилировать Java
- Требования: пакеты, железо, операционная система
- Как запускать на Skylake-X (Xeon и i9)
5. VMStructs: Why does an application need to know about JVM internals?
Andrey ( apangin ) is a person who collects the most profound and powerful reports over and over again. On the past Joker, he had a little less than a thousand people - a record in terms of audience size in a regular non-keyout report. In this he is helped by a decade of experience in virtual machines and the ability to explain technical hardcore so that it can be repeated in practice.
Many people do not quite understand why dig in a virtual machine if you have a regular application on a computer, with a database and everything is as it should be. In this report, there is a good argument at the level of "how to understand which query will pull a lot of data?" If you try to instrument the code with the help of JMX, then something strange is happening with the performance. Firstly, it is possible to understand why this is happening, secondly, what can be done with this. In my opinion, this report is valuable not so much a set of tools (although there is a little about helfy), as a demonstration of the correct mindsets developer OpenJDK and how to behave in difficult situations. The presence is spoken and explained the meaning of specific things like TLAB, Code Cache, Constant Pool, etc.
4. Korutin in Kotlin
Only the lazy one does not know about Kotlin now, and you, the reader, once read to the fourth point - obviously not one of the lazy. Roman ( elizarov ) - in the past, the developer of high-performance trading software, and now - Timlid in Kotlin libraries. We have already done an interview with Roma about Korutin , and it may be worth rereading it before or after viewing the report. Korutins are a very old concept, from the time of Simula, but not all mainstream languages support them, they will not appear in Java soon. But in Kotlin they are already in a stable version.
This report answers all topical questions about Korutin, that is, all topical issues of our time in general :-)
- Общая картина развития языков
- Асинхронное программирование с коллбэками
- Корутины в Kotlin
- regular loops, exception handling, higher-order functions
- custom higher-order functions
- все выглядит как в блокирующем коде!
- Как это работает?
- suspending functions, code with suspension points
- Retrofit async
- Коллбэки и континуации (call/cc из Scheme!)
- jdk, guava, nio, reactor, rx1, rx2
- Как запускать корутины?
- async / await
- Почему в Kotlin нет ключевого слова await?
- Concurrency — это сложно. Нужно делать это в явном виде.
- Подход Kotlin к async
- Концепция корутин: очень легкие треды
- Интероп с Java
- Синхронные корутины — generate/yield
- Пример с числами Фибоначчи
- Communicating Sequential Processes (CSP)
- Library vs Language
- Ядро языка должно быть небольшим!
- Keywords: async/await, generate/yield
- Modifiers: suspend
- kotlinx-coroutines: launch, async, runBlocking, future, delay, Job, Deferred, ...
3. On the shoulders of giants: the languages in which Kotlin studied
A report about Kotlin from one of the creators of the language - what else is needed for happiness? The essence and structure of the presentation is absolutely not the same as in the previous report of Elizarov. Roman talked about specific things - what, how and why in the design of korutin, and how to use it, and I need it to improve my programming skills on Kotlin. Here, Andrew ( abreslav ) tells things that improve knowledge of life in general and give an understanding of their place in the world. If you have seen all these languages with your own eyes - Java, C #, Scala, Groovy, Python, Gosu - this is even more interesting, because there is a reason for discussion (conference visitors probably could have taken the opportunity and really discussed their understanding of things live in the discussion area ). This is "seven languages in seven weeks.", but only in one hour.
By the way, we recently did a separate interview with Andrey , but it was not only about Kotlin, but about a lot of different things.
2. The Adventures of Senor Holmes and Junior Watson in the world of software development
Making reports in which more than one speaker participates is extremely difficult. Often it looks like this: half of the time one of them stands on the stage and is bored, and it looks very sad. What can be said about the joint performance of Baruch Sadogursky and Evgeny Borisov ( EvgenyBorisov ) - on the contrary, it was made magnificently, it is a work of art. The stars came together in the right order, and two top speakers appeared on the stage with a lot of experience and practice in conducting paired reports to discuss a topic of interest to both of them. Why I emphasize it this way - usually, viewers have no idea what the creation of such a presentation is worth, and take everything for granted.
But the result justified the investment. Judge for yourself, this is an evening keyout, the most recent report of the conference, during which people are already exhausted by two days of tin, they want to sleep, someone needs to fly home right now and so on. And yet, in the hall there are more than 600 people for sure.
This is not a reference report, but a report-show, you need to watch it. In it, Holmes and Watson will uncover several riddles that you have encountered, encountered or will encounter in everyday development. There will be no gutters of garbage collectors and bytecode, but there will be tools, libraries and frameworks that perplex ordinary developers in their daily routines, leading to downtime, deadlines and long-term depressions. Practically, in this report, Sherlock and Watson save your forehead from facepalms and rakes that someone has already attacked.
Therefore, the summary content here again will not be - just start the video and watch.
1. Boot yourself, Spring is coming
Baruch + Gamov, Baruch + Borisov, who is missing among the authors of the popular pair reports? That's right, Borisov + Tolkachev ( tolkkv ). Another cool report that, without being a keyout, gathered a record number of participants - more than a thousand. The material turned out to be so much, and it is so interesting that they allocated as many as two slots in the conference program - and you’ll have to watch two whole videos on YouTube, respectively.
Many years ago, Java programmers used “new” to create services. They did a lot of manual actions and mixed configuration with business logic. They even used copy-paste techniques. It was written many lines of poor code, which sometimes even worked.
Then Spring appeared. With him, much has changed ... We got a lot of "magic" from the magic cylinder of Spring, and our code has become more clean, simple and maintained.
And here was Spring Boot. On the one hand, it solves thousands of previously existing problems: version conflicts, configuration tasks, work with infrastructure bins, the problem of setting up the environment and, of course, launching or deploying the application, including building jar / war archives. On the other hand, Spring Boot added even more magic to our magic cylinder. As a result, there are two scenarios:
- Everything works great, although no one knows how.
- Nothing works, and no one knows why.
This report reveals the basic secrets of Spring Boot magic. You understand the basic principles and conventions by which typical Spring Boot applications work. The task of Cyril and Eugene is that all the magic from the cylinder turns into a much more transparent process for you and you can not only have fun when everything works, but also understand the essence of the problems that arise and can solve them without using ambulance and other services available urgent calls.
It took me a few days to watch all the videos from the list. It was not easy, but it is clear that it was worth it. In reality, there were much more reports at the conference, and watching the top ten of them was a good start for a much larger way. If you also watch these reports - do not be lazy to write your reviews in the comments on Habré!
In the meantime, it is already possible to purchase tickets for the next JPoint. It will be held April 5-6, 2019 at the WTC Congress Center. Until January 1, there is still an opportunity to purchase tickets at low prices !