August 21, 2017 at 14:03“Java developers are not aware of the problem with profilers”: Andrei Pangin and Nitsan Wakart on Java profiling
It’s easy to think that you should not expect big news from profiling: since developers have been profiling for decades, what could you think of there? But in Java profiling, there are serious pitfalls like safepoint bias, and new tools are emerging to solve such problems.
Andrey apangin Pangin (Odnoklassniki) recently created one of these tools: at the JPoint conference in April, he presented the open source project async-profiler , which safepoint bias is not afraid of. Another expert on this topic is Nitsan Wakart : many learned about the same problem with safepoints thanks to his blog post Why (Most) Sampling Java Profilers Are Fucking Terrible.
We decided to talk to both of them immediately, starting the conversation with the latest news about async-profiler, and later moving on to the state of Java profiling in general.
JUG.ru: Recently, async-profiler was moved to GitHub from Andrey’s personal repository in “jvm-profiling-tools” - what is jvm-profiling-tools, and what caused the move?
Nitsan: This is a combination of several repositories with similar goals. The idea is to gather them in one place and stimulate the community to develop them more actively.
Andrew: Exactly. When it became clear that the async-profiler is interesting to the Java community, I decided to move it to a neutral place, because it is sometimes uncomfortable for third-party developers to smuggle it into someone else's personal repository.
Nitsan:The jvm-profiling-tools have Honest Profiler , async-profiler and perf-map-agent , and they approach the same topic from different perspectives: perf-map-agent opens up profiling options using Linux perf, Honest Profiler uses AsyncGetCallTrace, which avoids safepoint bias, and async-profiler combines both in a very pleasant way.
Andrew: Yes. In fact, there is nothing cosmic in the idea of combining both approaches, but for some reason it had never occurred to anyone before.
Nitsan: Actually it was in Solaris Studio, but the problem with Solaris Studio is that it is used by about 20 people in the world.
Andrew: But, as far as I know, it does not show kernel calls, right?
Nitsan:Shows native code, but not core.
JUG.ru: Since Honest Profiler and async-profiler have the common advantage of “lack of safepoint bias”, now when looking at jvm-profiling-tools, newcomers to profiling may ask: “Well, which of two similar tools should I use?” What can you tell them?
Andrew:I believe that with regard to the accuracy of profiling and the completeness of information, async-profiler is far ahead. Indeed, even AsyncGetCallTrace in HotSpot does not always work: in some borderline cases, the JVM cannot restore the stack trace, although async-profiler also copes with such situations. In addition, the Honest Profiler does not show native stack traces at all. But its big advantage in infrastructure around data presentation. It can display results beautifully, it has a UI, and async-profiler is just a Java agent running from the console.
Nitsan:I think the future is for async-profiler. And I would like some of the Honest Profiler features to fall into async-profiler. There is still a difference in the fact that Honest Profiler runs on macOS, but async-profiler does not yet exist: maintaining a large community of hipsters in our industry is important.
JUG.ru: It seems that Vadim Tsesko from Odnoklassniki already made async-profiler possible on macOS?
Andrew: That was before I added perf events support. Linux-specific call broke macOS support. But there is good news: just the other day I was talking with Norman Maurer from Apple (author of Netty), he was also interested in async-profiler, and he kindly agreed to make a Mac port.
JUG.ru: In July, a hip profiler appeared in async-profiler - can you talk about this?
Andrew:There are two main approaches to memory profiling in Java. The first is bytecode instrumentation. But for production systems, it is terrible, because it has a deplorable effect on performance. A number of compiler optimizations stop working: first of all, Escape Analysis no longer helps to avoid allocations in heap.
Another approach is the use of DTrace probes, which is also extremely expensive, and can only be enabled at the start of the JVM.
But that's not all. There is a much more efficient approach based on TLAB (Thread Local Allocation Buffer) sampling. It is implemented in the Java Mission Control / Java Flight Recorder, but requires the inclusion of commercial Oracle JDK functions, and does not work with OpenJDK at all. A similar method is used internally by Google, but it requires the assembly of a modified version of the JVM.
I found a way to use this approach without connecting paid features, including with OpenJDK. Now I will not dive into the details, but I will definitely tell you in a separate report.
Nitsan: I think this is important. Since Java Mission Control is now perhaps the only tool for profiling allocations, and working with similar processes in JMC is implemented in a very peculiar way, many people simply do not do such profiling. Hope this helps profile profiling allotments to become mainstream.
JUG.ru: It may seem strange that noticeable shifts in profiling take place in 2017, when they would have been useful years ago. What is the reason for this delay?
Andrew:Java is the root of all evil :) It makes life both better and worse. On the one hand, due to the nature of the JVM, standard approaches are not applicable, but on the other, the JVM provides its own profiling APIs.
Nitsan: I think the Java world is like Windows. Windows was a terrible OS (probably much better now), but, suffering from many shortcomings, it was very successful at the same time. Same story with Java. In the case of profiling tools, Java did poorly. I am not quite sure why this is so.
I think that JVM developers have traditionally used Solaris Studio, so everything worked more or less normally, but only for them. Specialists used specialized tools. And most Java developers were happy with what they had.
But now Java had to face reality. Native profiling, which worked in Solaris Studio but was a niche solution then, is becoming increasingly popular.
Andrew: I consider it necessary to add that Java is not equal to HotSpot, and other JVMs may be more friendly to profilers.
Nitsan: Maybe I missed something, but what kind of JVM are we talking about? I know a lot about Zing, and I have a bit of experience with the IBM J9 ...
Andrew: Now I have an Excelsior JET shirt on, and so I remembered about this project. He knows how to precompile Java into native code, and, as far as I know, he does not suffer from safepoint bias.
Nitsan: Ah, approx. Never used it. I believe that in this case you can immediately take the native profiler.
JUG.ru: AsyncGetCallTrace, which is used by async-profiler and Honest Profiler, is not an official API. Is its use perceived as a “hack"? Are you worried that in the future it may stop functioning? Would more formal support for profiling help Oracle?
Nitsan: AsyncGetCallTrace has been working since the launch of OpenJDK 6, so it looks like it has always been and will work. This is an "illegitimate child", but I do not think that they can take it and throw it away. When something becomes a JVM option, it gets official support in some way. So I think we should not worry too much about this. Although I wonder how well AsyncGetCallTrace gets along with the new Graal compiler.
Of course, more formal support would help. Oracle currently provides JMC as a paid option, and the rest of the Java world is left with something like VisualVM. This is the money now. I think that Oracle is in a conflict of interests: Java interests, on the one hand, and their own, on the other. We can say that for them to contribute to the improvement of other JVM profilers would mean to worsen their own situation.
I do not claim that they are actively worried about this. I have no idea what they want now. Perhaps they will make things like the JMC available to everyone. Now that the work on Jigsaw is finished, they have a lot of time for another.
Andrew: I agree that AsyncGetCallTrace is partly a “hacker" API. Moreover, far from ideal: I myself reportedabout bugs. But so far this is the best that is in the HotSpot JVM.
JUG.ru: And can, besides existing things like AsyncGetCallTrace, something else appear that will make life easier for creators of profilers?
Andrew: Yes. HotSpot has recently discussed allocation profiling. In the end, there was even a JEP project that offers a new standardized API for heap sampling. I think that raising such topics in the mailing lists and offering JEP is the right way. So maybe someday in Java 11 ...
JUG.ru: What do you think of the future of Java profiling regardless of Oracle? Will profilers get much better than they are now?
Andrew:I really hope for it. In my reports, I try to convey to the developers the idea that the profilers that they use now have big pitfalls. It is necessary either to stop using them, or to properly understand their shortcomings and traps. And I think when more people realize the scale of the problem, the developers of these tools will begin to improve them.
Nitsan: In a number of issues there is room to grow. With perf-map-agent, we gained the ability to track inlining during profiling, but going to async-profiler, we lose it again. I would love to see her embodied again.
Another area is visualization. If you use async-profiler when working with multi-threaded applications, where one thread uses 100% of the CPU and all the others just hang on hold, you can get a confusing picture when profiling. I am interested in data presentation problems, and I am sure that there are many similar problems.
Andrew: Yes. Today FlameGraph is very popular as a visualization, but I would say that it is far from perfect.
JUG.ru: You both said (in reports and blog posts) that profilers can create a distorted picture. Do you think that the industry harms people to use this distorted picture? Could it be that industries from profilers then do more harm than good?
Nitsan:Yes, I think it harms the industry. If you look at the Internet about Java performance discussions, you’ll see that there’s plenty of buzz. And the reason that there is so much of it is because the information is difficult to verify.
Some people told me that the problem never ends up in the HashMap. And the reason they never thought that HashMap could be a problem is because a regular profiler will never show them that. I do not claim that it is in their case that the problem is in it, but be that as it may, they will never know about it. And when they look at the CPU timeline, they cannot see the time spent on the GC. That is, in the event that they had a “bottleneck” GC, they will not be able to track this.
Andrew:I agree with Nitsan, but I add that it is better to have at least a bad tool than not to have any. The main problem is not when the profiler is cheating, but when it is not used at all. Many developers do not profile at all, although often the performance problem lies in inefficient algorithms, and any sampling profiler will easily identify it.
Nitsan: I agree that many people do not profile, and this is a problem. But if your only tool turns out to be a bad profiler ... you look at it, see that it gives out some kind of nonsense, and say to the rest: “It’s better we just put timestamps”, so you get the conclusion, and it’s easy to understand how similar discourages people from using profilers.
JUG.ru: So, in order to improve the situation with Java profiling, we all need to work not only on improving tools, but also on community knowledge about them?
Nitsan: Yes. I think a large-scale success story in the Java world is JMH, Java Microbenchmarking Harness. And the reason is that this solution was very successful not only from a technological point of view, but also in terms of educating users, giving them the opportunity to better get acquainted with this area.
I think that Andrew creates tools and tells people is very important.
Andrew:Even the most powerful tools will be useless if you do not know how to use them. With the same JMH: I saw many times how people wrote nonsense in JMH, and then made completely wrong conclusions. Learning is an integral part of success.
JUG.ru: Both of you are going to help this training with your reports at the nearest Joker, and both of you are going to present new versions of reports presented earlier at other conferences there. What will be the novelty?
Nitsan: After I Introducedon QCon's “Profilers are lying hobbitses” report, I thought the best thing about it was the name. Therefore, I decided to keep the title, but the report itself will be very different. We will again talk about profilers and how they can mislead us, but I think I will start with the darkest, and then I will show how to get out. In the past, the report was a series of surprises leading to the conclusion “nothing works.” This time it will be like this: "nothing works, but let's see how we can deal with this."
Andrew:Initially, I planned to show the continuation of the history of async-profiler, which began at JPoint 2017. However, then the Program Committee and I discovered a great similarity between my report and the Nitsan report, so I decided to take a new topic. So far I’m not ready to say what exactly this will be, but in the near future in the Joker 2017 program you will see my new report! So stay tuned.
Joker Java conference , where Nitsan and Andrey will speak, will be held in St. Petersburg on November 3-4. As usual, after the reports, Joker speakers find themselves in discussion areas, so that there you can ask them about profiling personally. And besides Andrey and Nitsan, there will be dozens of other speakers - on the conference website you can see the program (and purchase a ticket).
