How to analyze Thread Dump
- Transfer
There are a lot of topics on the internals of the JVM in the Java Developer course . We understand the mechanisms of collections, bytecode, garbage collection, etc. Today we offer your heed a translation of a rather interesting article on thread dump. What it is, how to get it and how to use it.
Want to know how to analyze a thread dump? Come under the cat to learn more about how to get thread dump in Java and what to do with it.
Most modern Java applications are multi-threaded. Multithreading can significantly expand the functionality of the application, at the same time it introduces significant complexity.
In a single-threaded application, all resources (shared memory, I / O operations, etc.) can be used without synchronization, because at any given time only one thread is using the resource.
In the case of multi-threaded applications, it is necessary to find a compromise between the complexity of the program and a possible performance improvement, when several threads can use all available (often more than one) cores of the central processing unit (CPU). If you do it right, then using multithreading (formalized in Amdahl's Law), you can achieve a significant increase in application performance. However, it must be remembered about ensuring synchronous access of several threads to a shared resource. In most cases, frameworks such as Spring encapsulate threading and hide many technical details from users. However, in the case of the use of modern complex frameworks, something can go wrong, and we, as users, will encounter difficult-to-solve multi-threading bugs.
Fortunately, Java is equipped with a special mechanism for obtaining information about the current state of all threads at any given time - this is a thread dump (a kind of snapshot). In this article, we will learn how to get a thread dump for an application of realistic dimensions and how to analyze this dump.
It is assumed that the reader has basic information about multi-threaded programming and is aware of the problems of thread synchronization and the use of shared resources. Nevertheless, it will not be superfluous to brush up on some basic terms and concepts.
At first glance, Java thread dumps may seem like a “Chinese diploma”, the key to which I understand is the following concepts. In general, let's repeat the main terms of multithreading, which we will use for the analysis of dumps.
The terminology presented is not exhaustive for describing the world of multithreading, but this is enough to begin analyzing thread dumps.
More detailed information can be found in these sources:
Section 17 of the JLS and Java Concurrency in Practice
By applying these simple notions about flow in Java, we can create a test application. For this application we will collect thread dump. The resulting dump will analyze and extract useful information about the current threads of the application.
Before we create a thread dump, we need to develop a Java application. The traditional “hello, world!” Is too simple for our purpose, and a medium sized application dump may be too difficult to demonstrate. Based on this, we will create a fairly simple application in which two streams are created. And the threads get into the deadlock:
This program creates two resources: resourceA and resourceB, and starts two threads: threadLockingResourceAFirst and threadLockingResourceBFirst, which block each other’s resources.
The cause of deadlock is a “cross” blocking of resources by threads.
The cause of the deadlock is an attempt to "mutual" capture of resources, i.e. thread threadLockingResourceAFirst captures resource resourceA, thread threadLockingResourceBFirst captures resource resourceB. After that, the thread threadLockingResourceAFirst, without releasing its resource, tries to capture resourceB, and the thread threadLockingResourceBFirst, without releasing its resource, tries to capture resource ResourceA. As a result, threads are blocked. A 1s delay is added to ensure blocking occurs. Streams are waiting for the release of the necessary resources, but this will never happen.
The output of the program will be as follows (the numbers after java.lang.Object @ will be different for each launch):
After the output of these messages, the program will look like working (the process that executes this program is not completed), while the program does not perform any work. This is how deadlock looks like in practice. To solve the problem, we need to manually create a tread dump and analyze the state of the threads.
In practice, a Java program can crash and create a thread dump. However, in some cases (for example, in the case of deadlocks), the program does not terminate and thread dump does not create, it just hangs. To create a dump of such hung programs, first of all you need to find out the identifier of the program process, i.e. Process ID (PID). To do this, you can use the JVM Process Status (JPS) utility, which, since version 7, is part of the Java Development Kit (JDK). To find the PID process of our hung program, we simply execute jps in the terminal (Windows or Linux):
The first column is the local virtual machine identifier (Local VM ID, i.e. lvmid) for the running Java process. In the context of the local JVM, lvmid points to the PID of the Java process.
It should be noted that this value is likely to differ from the value above. The second column is the name of the application, which may indicate the name of the main class, jar-file or be equal to "Unknown". It all depends on how the application was launched.
In our case, the application name DeadlockProgram is the name of the main classes that was launched when the program started. In the example above, the PID of the program is 11568, this information is enough to generate a thread dump. To generate a dump, we will use the jstack utility , which is included in the JDK, starting with version 7. To get the dump, we will transfer tojstack as the PID parameter of our program and specify the -l flag (create a long listing). The utility output is redirected to a text file, i.e. thread_dump.txt:
The resulting thread_dump.txt file contains a thread dump of our hung program and contains important information for diagnosing the causes of the deadlock.
If you are using JDK up to version 7, then you can use the Linux utility - kill with the -3 flag to generate a dump . Calling kill -3 will send a SIGQUIT signal to the program.
In our case, the call will be like this:
Opening the file thread_dump.txt, we will see something like the following content:
Although at first glance this file may seem too complicated and confusing, in reality it is very simple, if you take it apart in parts, step by step.
The first line indicates the time when the dump was generated, the second - the diagnostic information about the JVM, on which the dump was received:
There is no stream information in this section. This sets the general context of the system in which the dump was collected.
The next section provides information about the threads that were running in the system at the time of the dump:
The following section provides a list:
Safe Memory Reclamation (SMR) information
It contains information about streams outside of the JVM, i.e. these are not virtual machine threads or garbage collector threads. If you look at the addresses of these threads, you can see that they correspond to the value of tid - the “natural, iron” (native) address in the operating system, and not the Thread ID.
Triplets are used to hide unnecessary information:
Immediately after the SMR block follows a list of threads. The first thread on our list is the Reference Handler:
The first line for each stream provides a general description. The description contains the following items:
The second line is the current state of the stream. Possible flow states are listed in enum:
Thread.State :
NEW
RUNNABLE
BLOCKED
WAITING
TIMED_WAITING
TERMINATED
For more information, see the documentation .
The next section contains the stack trace trace at the time of the dump. This stack trace is very similar to the trace that is generated by an uncaught exception. It contains the names of the classes and lines that were executed at the time of the dump. In the case of the Reference Handler stream, we see nothing interesting.
However, there is something interesting in the trace of Thread-02 that differs from the standard trace:
In the trace, we see that information about the blocking has been added. This thread is waiting for a lock on the object with the address 0x00000000894465b0 (object type java.lang.Object). Moreover, the thread itself holds the lock with the address 0x00000000894465a0 (also a java.lang.Object object). This information is useful to us further for the diagnosis of deadlock.
The last section provides a list of synchronization primitives captured by the stream. These are objects that can be used to synchronize threads, for example, locks.
According to official Java documentation, Ownable Synchronizer is the heirs of the AbstractOwnableSynchronizer (or its subclass) that can be exclusively captured by the stream for synchronization purposes.
ReentrantLock and write-lock , but not the read-lock class of the ReentrantReadWriteLock are two good examples of such “ownable synchronizers” offered by the platform.
For more information on this issue, please refer to this
post..
The next section of the dump contains information about JVM technical threads that are not part of the application and are associated with operating system threads. Because these threads work outside the application, they do not have thread identifiers. Most often these are garbage collector streams and other technical JVM streams:
This section indicates the number of global links used by the JVM through the JNI. These links are not served by the garbage collector and in certain circumstances may cause a memory leak.
In most simple cases, this information is not used. However, the importance of global links must be understood. For more information, see this post .
The last section contains information about deadlocks found.
If these do not show up, then the section will be empty. Because we specifically developed the application with locks, in our case this section is there. A lock was detected during dump generation and is represented by the following message:
The first subsection describes the deadlock scenario:
Thread-0 expects the ability to capture the monitor (this is a call to the synchronized (secondResource) block in our application), while this thread holds the monitor trying to capture the Thread-1 flow ( This is a reference to the same code snippet: synchronized (secondResource) in our application.
This circular lock is otherwise called deadlock . In the figure below,
this situation is presented graphically:
In the second subsection, the stack trace is given for both blocked threads.
This stack trace allows us to follow the operation of each thread until a lock appears.
In our case, if we look at the line:
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34) , we will see the problematic piece of code:
This line is the first line of the synchronized block, which is the cause of the lock, and tells us that the synchronization on the secondResource is the cause of the interlocking. To remedy the situation, we must ensure the same synchronization order on resourceA and resourceB in both flows. If we do this, we will come to the following application:
This application will end without a deadlock, and as a result we will get the following output (note that the addresses of the Object class instances have changed):
As a result, using only the information obtained from the thread dump, we were able to find the cause and fix the deadlock in our application. The proposed technique is sufficient for most simple applications (or applications with a small number of deadlocks). To deal with more complex situations, will require other methods.
Dumps of real applications can be very large and complex.
In one JVM hundreds of thousands of threads can be running at the same time. And more than two streams may be blocked (or there may be several threading problems caused by one problem).
The analysis of such a huge amount of information can be a real problem.
For analyzing large dumps, special analyzer utilities are used - Thread Dump Analyzers (TDAs). These utilities parse Java thread dumps and display information in a human-readable form, often using graphical tools. Moreover, some of them can perform static analysis and find the cause of the problem. Of course, the choice of a particular utility depends on a number of circumstances.
Nevertheless, here is a list of the most popular TDAs:
This is not a complete list of utilities of this kind. Each utility from this list performs a sufficiently qualitative analysis and simplifies the manual labor of analyzing dumps.
Thread dumps are a great tool for analyzing the state of a Java application, especially in cases of unexpected behavior of multi-threaded applications. However, without the proper baggage of knowledge, dumps can add additional complexity to an already difficult situation.
In this article, we developed a deadlock application, dumped the hung application. Analyzed the dump and found the cause of the lock and fixed it. This is not always so easy, for the analysis of most real-world applications it is very useful to use special utilities - analyzers.
However, every professional Java developer needs to understand the basics of thread dump analysis. Should be guided in their structure, know what information can be extracted and how to use it to solve problems of multithreading.
Although thread dump is not a “silver bullet” in the multithreading world, it is nevertheless an important tool for diagnosing complex, but quite common problems of multi-threaded Java applications.
In the Java Developer course program, multi-threading issues occupy a noticeable part. We consider in detail how to develop programs so that we don’t have to deal with deadlock production at night.
As always, your opinions and comments are interesting, which you can leave here or have a look at Vitaly on an open day .
Want to know how to analyze a thread dump? Come under the cat to learn more about how to get thread dump in Java and what to do with it.
Most modern Java applications are multi-threaded. Multithreading can significantly expand the functionality of the application, at the same time it introduces significant complexity.
In a single-threaded application, all resources (shared memory, I / O operations, etc.) can be used without synchronization, because at any given time only one thread is using the resource.
In the case of multi-threaded applications, it is necessary to find a compromise between the complexity of the program and a possible performance improvement, when several threads can use all available (often more than one) cores of the central processing unit (CPU). If you do it right, then using multithreading (formalized in Amdahl's Law), you can achieve a significant increase in application performance. However, it must be remembered about ensuring synchronous access of several threads to a shared resource. In most cases, frameworks such as Spring encapsulate threading and hide many technical details from users. However, in the case of the use of modern complex frameworks, something can go wrong, and we, as users, will encounter difficult-to-solve multi-threading bugs.
Fortunately, Java is equipped with a special mechanism for obtaining information about the current state of all threads at any given time - this is a thread dump (a kind of snapshot). In this article, we will learn how to get a thread dump for an application of realistic dimensions and how to analyze this dump.
It is assumed that the reader has basic information about multi-threaded programming and is aware of the problems of thread synchronization and the use of shared resources. Nevertheless, it will not be superfluous to brush up on some basic terms and concepts.
Basic terminology
At first glance, Java thread dumps may seem like a “Chinese diploma”, the key to which I understand is the following concepts. In general, let's repeat the main terms of multithreading, which we will use for the analysis of dumps.
- Thread or thread is a discrete unit of multithreading driven by a Java Virtual Machine (JVM). JVM threads correspond to threads in the operating system (OS) - native threads (“natural flows”), which implement the mechanism for executing code.
Each thread has a unique identifier and name. Streams can be "demons" and "not demons."
The program ends its work when all non-daemons threads are terminated or the Runtime.exit method is called . Working "demons" do not affect the completion of the program. Those. JVMs are waiting for all the “non-demons” to be completed and the work is completed, and the “non-demons” pays no attention.
For more information, see the documentation for the class Thread .
A thread can be in one of the following states:- Alive thread or “live” is a thread that does some work (normal state).
- Blocked thread or “blocked” - a thread that tried to enter the synchronization section (synchronized), however, another thread has already managed to enter this block first, and all the following flows that try to enter this same block are blocked.
- Waiting thread or “waiting” is the thread that called the wait method (possibly with a timeout indication) and is now waiting for another method to execute notify or nonifyAll on the same object.
Note that the thread is not considered "waiting" if it has called wait with a timeout and this timeout has expired. - Sleeping thread or "sleeping" - a thread that is not currently running, because performed the Thread.sleep method (indicating the duration of the "sleep").
- Monitor is the mechanism used in the JVM to provide multi-threaded access to a single object. The mechanism is started using the special synchronized keyword .
Each object in Java has a monitor with which the stream can synchronize, i.e. set a lock, which guarantees that no other thread will gain access to this object until the lock is released, i.e. thread - lock owner will not exit synchronized block .
For more information, see the Synchronization section (17.1) of the Java Langauge Specification (JLS) . - Deadlock is a situation in which a thread, say A, blocks a resource, it needs another resource that is blocked by another thread, say B. Flow B does not release this resource, because to complete a certain operation, it needs a resource that is blocked by stream A. It turns out that stream A is waiting for the resource to be unlocked by thread B, which is waiting for another resource to be unlocked by stream A. And thus, the threads are waiting for each other. As a result, the whole program is "hanging" and waiting for the threads to somehow unlock and continue working. Mutual blocking can have many threads. This problem is well known as the “Problem of the Dining Philosophers” .
- Livelock is a situation where the flow A causes the flow B to perform some action, which in turn leads to the execution of the flow A of the original action, which once again causes the action of the flow B. A cyclical dependence is obtained. This can be imagined as a dog running after its tail. Similarly to Deadlock , in the Livelock situation the program does not make progress, i.e. does not perform a useful action, however, in this situation, threads are not blocked.
The terminology presented is not exhaustive for describing the world of multithreading, but this is enough to begin analyzing thread dumps.
More detailed information can be found in these sources:
Section 17 of the JLS and Java Concurrency in Practice
By applying these simple notions about flow in Java, we can create a test application. For this application we will collect thread dump. The resulting dump will analyze and extract useful information about the current threads of the application.
Creating an example program
Before we create a thread dump, we need to develop a Java application. The traditional “hello, world!” Is too simple for our purpose, and a medium sized application dump may be too difficult to demonstrate. Based on this, we will create a fairly simple application in which two streams are created. And the threads get into the deadlock:
publicclassDeadlockProgram{
publicstaticvoidmain(String[] args)throws Exception {
Object resourceA = new Object();
Object resourceB = new Object();
Thread threadLockingResourceAFirst = new Thread(new DeadlockRunnable(resourceA, resourceB));
Thread threadLockingResourceBFirst = new Thread(new DeadlockRunnable(resourceB, resourceA));
threadLockingResourceAFirst.start();
Thread.sleep(500);
threadLockingResourceBFirst.start();
}
privatestaticclassDeadlockRunnableimplementsRunnable{
privatefinal Object firstResource;
privatefinal Object secondResource;
publicDeadlockRunnable(Object firstResource, Object secondResource){
this.firstResource = firstResource;
this.secondResource = secondResource;
}
@Overridepublicvoidrun(){
try {
synchronized(firstResource) {
printLockedResource(firstResource);
Thread.sleep(1000);
synchronized(secondResource) {
printLockedResource(secondResource);
}
}
} catch (InterruptedException e) {
System.out.println("Exception occurred: " + e);
}
}
privatestaticvoidprintLockedResource(Object resource){
System.out.println(Thread.currentThread().getName() + ": locked resource -> " + resource);
}
}
}This program creates two resources: resourceA and resourceB, and starts two threads: threadLockingResourceAFirst and threadLockingResourceBFirst, which block each other’s resources.
The cause of deadlock is a “cross” blocking of resources by threads.
The cause of the deadlock is an attempt to "mutual" capture of resources, i.e. thread threadLockingResourceAFirst captures resource resourceA, thread threadLockingResourceBFirst captures resource resourceB. After that, the thread threadLockingResourceAFirst, without releasing its resource, tries to capture resourceB, and the thread threadLockingResourceBFirst, without releasing its resource, tries to capture resource ResourceA. As a result, threads are blocked. A 1s delay is added to ensure blocking occurs. Streams are waiting for the release of the necessary resources, but this will never happen.
The output of the program will be as follows (the numbers after java.lang.Object @ will be different for each launch):
Thread-0: lockedresource-> java.lang.Object@149bc794
Thread-1: locked resource -> java.lang.Object@17c10009After the output of these messages, the program will look like working (the process that executes this program is not completed), while the program does not perform any work. This is how deadlock looks like in practice. To solve the problem, we need to manually create a tread dump and analyze the state of the threads.
Thread Dump Generation
In practice, a Java program can crash and create a thread dump. However, in some cases (for example, in the case of deadlocks), the program does not terminate and thread dump does not create, it just hangs. To create a dump of such hung programs, first of all you need to find out the identifier of the program process, i.e. Process ID (PID). To do this, you can use the JVM Process Status (JPS) utility, which, since version 7, is part of the Java Development Kit (JDK). To find the PID process of our hung program, we simply execute jps in the terminal (Windows or Linux):
$ jps
11568 DeadlockProgram
15584 Jps
15636The first column is the local virtual machine identifier (Local VM ID, i.e. lvmid) for the running Java process. In the context of the local JVM, lvmid points to the PID of the Java process.
It should be noted that this value is likely to differ from the value above. The second column is the name of the application, which may indicate the name of the main class, jar-file or be equal to "Unknown". It all depends on how the application was launched.
In our case, the application name DeadlockProgram is the name of the main classes that was launched when the program started. In the example above, the PID of the program is 11568, this information is enough to generate a thread dump. To generate a dump, we will use the jstack utility , which is included in the JDK, starting with version 7. To get the dump, we will transfer tojstack as the PID parameter of our program and specify the -l flag (create a long listing). The utility output is redirected to a text file, i.e. thread_dump.txt:
jstack -l 11568 > thread_dump.txtThe resulting thread_dump.txt file contains a thread dump of our hung program and contains important information for diagnosing the causes of the deadlock.
If you are using JDK up to version 7, then you can use the Linux utility - kill with the -3 flag to generate a dump . Calling kill -3 will send a SIGQUIT signal to the program.
In our case, the call will be like this:
kill-311568Analysis of simple thread dump
Opening the file thread_dump.txt, we will see something like the following content:
2018-06-19 16:44:44
Full thread dump Java HotSpot (TM) 64-Bit VM Server (10.0.1 + 10 mixed mode):
Threads class SMR info:
_java_thread_list = 0x00000250e5488a00, length = 13, elements = {
0x00000250e4979000, 0x00000250e4982800, 0x00000250e52f2800, 0x00000250e4992800,
0x00000250e4995800, 0x00000250e49a5800, 0x00000250e49ae800, 0x00000250e5324000,
0x00000250e54cd800, 0x00000250e54cf000, 0x00000250e54d1800, 0x00000250e54d2000,
0x00000250e54d0800
}
"Reference Handler" # 2 daemon prio = 10 os_prio = 2 tid = 0x00000250e4979000 nid = 0x3c28 waiting on condition [0x000000b82a9ff000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList (java.base@10.0.1/Native Method)
at java.lang.ref.Reference.processPendingReferences (java.base@10.0.1/Reference.java: 174)
at java.lang.ref.Reference.access $ 000 (java.base@10.0.1/Reference.java: 44)
at java.lang.ref.Reference $ ReferenceHandler.run (java.base@10.0.1/Reference.java: 138)
Locked ownable synchronizers:
- None
"Finalizer" # 3 daemon prio = 8 os_prio = 1 tid = 0x00000250e4982800 nid = 0x2a54 in Object.wait () [0x000000b82aaff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait (java.base@10.0.1/Native Method)
- waiting on <0x0000000089509410> (a java.lang.ref.ReferenceQueue $ Lock)
at java.lang.ref.ReferenceQueue.remove (java.base@10.0.1/ReferenceQueue.java: 151)
- waiting for re-lock in wait () <0x0000000089509410> (a java.lang.ref.ReferenceQueue $ Lock)
at java.lang.ref.ReferenceQueue.remove (java.base@10.0.1/ReferenceQueue.java: 172)
at java.lang.ref.Finalizer $ FinalizerThread.run (java.base@10.0.1/Finalizer.java: 216)
Locked ownable synchronizers:
- None
"Signal Dispatcher" # 4 daemon prio = 9 os_prio = 2 tid = 0x00000250e52f2800 nid = 0x2184 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Attach Listener" # 5 daemon prio = 5 os_prio = 2 tid = 0x00000250e4992800 nid = 0x1624 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" # 6 daemon prio = 9 os_prio = 2 tid = 0x00000250e4995800 nid = 0x4198 waiting on condition [0x000000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
Locked ownable synchronizers:
- None
"C2 CompilerThread1" # 7 daemon prio = 9 os_prio = 2 tid = 0x00000250e49a5800 nid = 0x3b98 waiting on condition [0x000000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
Locked ownable synchronizers:
- None
"C1 CompilerThread2" # 8 daemon prio = 9 os_prio = 2 tid = 0x00000250e49ae800 nid = 0x1a84 waiting on condition [0x000000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
Locked ownable synchronizers:
- None
"Sweeper thread" # 9 daemon prio = 9 os_prio = 2 tid = 0x00000250e5324000 nid = 0x5f0 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Service Thread" # 10 daemon prio = 9 os_prio = 0 tid = 0x00000250e54cd800 nid = 0x169c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
Common-Cleaner # 11 daemon prio = 8 os_prio = 1 tid = 0x00000250e54cf000 nid = 0x1610 in Object.wait () [0x000000b82b2fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait (java.base@10.0.1/Native Method)
- waiting on <0x000000008943e600> (a java.lang.ref.ReferenceQueue $ Lock)
at java.lang.ref.ReferenceQueue.remove (java.base@10.0.1/ReferenceQueue.java: 151)
- waiting for re-lock in wait () <0x000000008943e600> (a java.lang.ref.ReferenceQueue $ Lock)
at jdk.internal.ref.CleanerImpl.run (java.base@10.0.1/CleanerImpl.java: 148)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
at jdk.internal.misc.InnocuousThread.run (java.base@10.0.1/InnocuousThread.java: 134)
Locked ownable synchronizers:
- None
"Thread-0" # 12 prio = 5 os_prio = 0 tid = 0x00000250e54d1800 nid = 0xdec waiting for monitor entry [0x000000b82b4ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465b0> (a java.lang.Object)
- locked <0x00000000894465a0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Locked ownable synchronizers:
- None
"Thread-1" # 13 prio = 5 os_prio = 0 tid = 0x00000250e54d2000 nid = 0x415c waiting for monitor entry [0x000000b82b5ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465a0> (a java.lang.Object)
- locked <0x00000000894465b0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Locked ownable synchronizers:
- None
"DestroyJavaVM" # 14 prio = 5 os_prio = 0 tid = 0x00000250e54d0800 nid = 0x2b8c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"VM Thread" os_prio = 2 tid = 0x00000250e496d800 nid = 0x1920 runnable
"GC Thread # 0" os_prio = 2 tid = 0x00000250c35b5800 nid = 0x310c runnable
"GC Thread # 1" os_prio = 2 tid = 0x00000250c35b8000 nid = 0x12b4 runnable
"GC Thread # 2" os_prio = 2 tid = 0x00000250c35ba800 nid = 0x43f8 runnable
"GC Thread # 3" os_prio = 2 tid = 0x00000250c35c0800 nid = 0x20c0 runnable
"G1 Main Marker" os_prio = 2 tid = 0x00000250c3633000 nid = 0x4068 runnable
"G1 Conc # 0" os_prio = 2 tid = 0x00000250c3636000 nid = 0x3e28 runnable
"G1 Refine # 0" os_prio = 2 tid = 0x00000250c367e000 nid = 0x3c0c runnable
"G1 Refine # 1" os_prio = 2 tid = 0x00000250e47fb800 nid = 0x3890 runnable
"G1 Refine # 2" os_prio = 2 tid = 0x00000250e47fc000 nid = 0x32a8 runnable
"G1 Refine # 3" os_prio = 2 tid = 0x00000250e47fd800 nid = 0x3d00 runnable
"G1 Young RemSet Sampling" os_prio = 2 tid = 0x00000250e4800800 nid = 0xef4 runnable
"VM Periodic Task Thread" os_prio = 2 tid = 0x00000250e54d6800 nid = 0x3468 waiting on condition
JNI global references: 2
Found one java-level deadlock:
=============================
"Thread-0":
monitor 0x00000250e4982480 (object 0x00000000894465b0, a java.lang.Object) waiting for lock
which is held by "Thread-1"
"Thread-1":
monitor 0x00000250e4982380 (object 0x00000000894465a0, a java.lang.Object) waiting for lock
which is held by "Thread-0"
Java stack information listed above:
================================================= =
"Thread-0":
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465b0> (a java.lang.Object)
- locked <0x00000000894465a0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
"Thread-1":
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465a0> (a java.lang.Object)
- locked <0x00000000894465b0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Found 1 deadlock.
Introductory Information
Although at first glance this file may seem too complicated and confusing, in reality it is very simple, if you take it apart in parts, step by step.
The first line indicates the time when the dump was generated, the second - the diagnostic information about the JVM, on which the dump was received:
2018-06-19 16:44:44FullthreaddumpJavaHotSpot(TM) 64-BitServerVM (10.0.1+10 mixedmode):There is no stream information in this section. This sets the general context of the system in which the dump was collected.
General flow information
The next section provides information about the threads that were running in the system at the time of the dump:
Threads class SMR info:
_java_thread_list = 0x00000250e5488a00, length = 13, elements = {
0x00000250e4979000, 0x00000250e4982800, 0x00000250e52f2800, 0x00000250e4992800,
0x00000250e4995800, 0x00000250e49a5800, 0x00000250e49ae800, 0x00000250e5324000,
0x00000250e54cd800, 0x00000250e54cf000, 0x00000250e54d1800, 0x00000250e54d2000,
0x00000250e54d0800
}
The following section provides a list:
Safe Memory Reclamation (SMR) information
It contains information about streams outside of the JVM, i.e. these are not virtual machine threads or garbage collector threads. If you look at the addresses of these threads, you can see that they correspond to the value of tid - the “natural, iron” (native) address in the operating system, and not the Thread ID.
Triplets are used to hide unnecessary information:
"Reference Handler" # 2 ... tid = 0x00000250e4979000 ... "Finalizer" # 3 ... tid = 0x00000250e4982800 ... "Signal Dispatcher" # 4 ... tid = 0x00000250e52f2800 ... "Attach Listener" # 5 ... tid = 0x00000250e4992800 ... "C2 CompilerThread0" # 6 ... tid = 0x00000250e4995800 ... "C2 CompilerThread1" # 7 ... tid = 0x00000250e49a5800 ... "C1 CompilerThread2" # 8 ... tid = 0x00000250e49ae800 ... "Sweeper thread" # 9 ... tid = 0x00000250e5324000 ... "Service Thread" # 10 ... tid = 0x00000250e54cd800 ... Common Cleaner # 11 ... tid = 0x00000250e54cf000 ... "Thread-0" # 12 ... tid = 0x00000250e54d1800 ... "Thread-1" # 13 ... tid = 0x00000250e54d2000 ... "DestroyJavaVM" # 14 ... tid = 0x00000250e54d0800 ...
Streams
Immediately after the SMR block follows a list of threads. The first thread on our list is the Reference Handler:
"Reference Handler" # 2 daemon prio = 10 os_prio = 2 tid = 0x00000250e4979000 nid = 0x3c28 waiting on condition [0x000000b82a9ff000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList (java.base@10.0.1/Native Method)
at java.lang.ref.Reference.processPendingReferences (java.base@10.0.1/Reference.java: 174)
at java.lang.ref.Reference.access $ 000 (java.base@10.0.1/Reference.java: 44)
at java.lang.ref.Reference $ ReferenceHandler.run (java.base@10.0.1/Reference.java: 138)
Locked ownable synchronizers:
- None
Brief stream description
The first line for each stream provides a general description. The description contains the following items:
| Section | Example | Description |
|---|---|---|
| Name | "Reference Handler" | The human-readable name of the thread. Name can be set by calling the setName object Thread A . And get through a call to getName |
| ID | # 2 | Unique ID assigned to each object of class Thread . ID is generated for threads in the system. The initial value is 1. Each newly created thread is assigned its own ID, previously increased by 1. This property of the “read-only” stream can be obtained using the getId function of the object of the class Thread . |
| Daemon status | daemon | The flag is a sign that the thread is a daemon. If it is a demon, then the flag will be set. For example, Thread-0 is not a daemon. |
| Priority | prio = 10 | The numeric priority of the java stream. Note that this priority does not necessarily correspond to the priority of the associated thread in the operating system. To set the priority, you can use the setPriority method of the Thread class object , and to get the getPriority method . |
| OS Thread Priority | os_prio = 2 | The priority of the thread in the operating system. This priority may differ from the one assigned to the coherent java stream. |
| Address | tid = 0x00000250e4979000 | Java stream address This address is a pointer to a Java Native Interface (JNI) native object of class Thread (a C ++ Thread object that is associated with a Java stream via JNI). This value is obtained by casting a pointer to this (a C ++ object that is associated with this Java stream) to an integer. See line 879 in hotspot / share / runtime / thread.cpp : st-> print ("tid =" INTPTR_FORMAT "", p2i (this));
Although the key for this object ( tid ) may be similar to a thread ID, in fact, this is the address of a connected JNI C ++ Thread object , and this is not the value that the getId method of the Java Thread object returns . |
| OS Thread ID | nid = 0x3c28 | The unique identifier for the operating system thread to which the Java thread is bound. This value is output by the following code: line 42 in hotspot / share / runtime / osThread.cpp : st-> print ("nid = 0x% x", thread_id ());
|
| Status | waiting on condition | The human-readable status of the current thread. This line prints additional information to the simple status of the thread (see below), which can be used to understand what the thread was going to do (i.e., whether the thread tried to block or waited for the unblocking condition). |
| Last Known Java Stack Pointer | [0x000000b82a9ff000] | The last known pointer to the stack (SP) associated with this thread. This value is obtained using native C ++ code mixed with Java code using JNI. The value returned by the function last_Java_sp () , line 2886 in hotspot / share / runtime / thread.cpp : st-> print_cr ("[" INTPTR_FORMAT "]",
(intptr_t) last_Java_sp () & ~ right_n_bits (12));
For simple thread dumps, this information is almost useless. However, in difficult cases, SP can be used to track locks. |
Flow condition
The second line is the current state of the stream. Possible flow states are listed in enum:
Thread.State :
NEW
RUNNABLE
BLOCKED
WAITING
TIMED_WAITING
TERMINATED
For more information, see the documentation .
Thread stack trace
The next section contains the stack trace trace at the time of the dump. This stack trace is very similar to the trace that is generated by an uncaught exception. It contains the names of the classes and lines that were executed at the time of the dump. In the case of the Reference Handler stream, we see nothing interesting.
However, there is something interesting in the trace of Thread-02 that differs from the standard trace:
"Thread-0" # 12 prio = 5 os_prio = 0 tid = 0x00000250e54d1800 nid = 0xdec waiting for monitor entry [0x000000b82b4ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465b0> (a java.lang.Object)
- locked <0x00000000894465a0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Locked ownable synchronizers:
- None
In the trace, we see that information about the blocking has been added. This thread is waiting for a lock on the object with the address 0x00000000894465b0 (object type java.lang.Object). Moreover, the thread itself holds the lock with the address 0x00000000894465a0 (also a java.lang.Object object). This information is useful to us further for the diagnosis of deadlock.
Captured synchronization primitives (Ownable Synchronizer)
The last section provides a list of synchronization primitives captured by the stream. These are objects that can be used to synchronize threads, for example, locks.
According to official Java documentation, Ownable Synchronizer is the heirs of the AbstractOwnableSynchronizer (or its subclass) that can be exclusively captured by the stream for synchronization purposes.
ReentrantLock and write-lock , but not the read-lock class of the ReentrantReadWriteLock are two good examples of such “ownable synchronizers” offered by the platform.
For more information on this issue, please refer to this
post..
JVM threads
The next section of the dump contains information about JVM technical threads that are not part of the application and are associated with operating system threads. Because these threads work outside the application, they do not have thread identifiers. Most often these are garbage collector streams and other technical JVM streams:
"VM Thread" os_prio = 2 tid = 0x00000250e496d800 nid = 0x1920 runnable "GC Thread # 0" os_prio = 2 tid = 0x00000250c35b5800 nid = 0x310c runnable "GC Thread # 1" os_prio = 2 tid = 0x00000250c35b8000 nid = 0x12b4 runnable "GC Thread # 2" os_prio = 2 tid = 0x00000250c35ba800 nid = 0x43f8 runnable "GC Thread # 3" os_prio = 2 tid = 0x00000250c35c0800 nid = 0x20c0 runnable "G1 Main Marker" os_prio = 2 tid = 0x00000250c3633000 nid = 0x4068 runnable "G1 Conc # 0" os_prio = 2 tid = 0x00000250c3636000 nid = 0x3e28 runnable "G1 Refine # 0" os_prio = 2 tid = 0x00000250c367e000 nid = 0x3c0c runnable "G1 Refine # 1" os_prio = 2 tid = 0x00000250e47fb800 nid = 0x3890 runnable "G1 Refine # 2" os_prio = 2 tid = 0x00000250e47fc000 nid = 0x32a8 runnable "G1 Refine # 3" os_prio = 2 tid = 0x00000250e47fd800 nid = 0x3d00 runnable "G1 Young RemSet Sampling" os_prio = 2 tid = 0x00000250e4800800 nid = 0xef4 runnable "VM Periodic Task Thread" os_prio = 2 tid = 0x00000250e54d6800 nid = 0x3468 waiting on condition
JNI global links
This section indicates the number of global links used by the JVM through the JNI. These links are not served by the garbage collector and in certain circumstances may cause a memory leak.
JNI global references: 2
In most simple cases, this information is not used. However, the importance of global links must be understood. For more information, see this post .
Deadlocked streams
The last section contains information about deadlocks found.
If these do not show up, then the section will be empty. Because we specifically developed the application with locks, in our case this section is there. A lock was detected during dump generation and is represented by the following message:
Found one java-level deadlock:
=============================
"Thread-0":
monitor 0x00000250e4982480 (object 0x00000000894465b0, a java.lang.Object) waiting for lock
which is held by "Thread-1"
"Thread-1":
monitor 0x00000250e4982380 (object 0x00000000894465a0, a java.lang.Object) waiting for lock
which is held by "Thread-0"
Java stack information listed above:
================================================= =
"Thread-0":
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465b0> (a java.lang.Object)
- locked <0x00000000894465a0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
"Thread-1":
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465a0> (a java.lang.Object)
- locked <0x00000000894465b0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Found 1 deadlock.
The first subsection describes the deadlock scenario:
Thread-0 expects the ability to capture the monitor (this is a call to the synchronized (secondResource) block in our application), while this thread holds the monitor trying to capture the Thread-1 flow ( This is a reference to the same code snippet: synchronized (secondResource) in our application.
This circular lock is otherwise called deadlock . In the figure below,
this situation is presented graphically:
In the second subsection, the stack trace is given for both blocked threads.
This stack trace allows us to follow the operation of each thread until a lock appears.
In our case, if we look at the line:
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34) , we will see the problematic piece of code:
printLockedResource (secondResource);
This line is the first line of the synchronized block, which is the cause of the lock, and tells us that the synchronization on the secondResource is the cause of the interlocking. To remedy the situation, we must ensure the same synchronization order on resourceA and resourceB in both flows. If we do this, we will come to the following application:
publicclassDeadlockProgram{
publicstaticvoidmain(String[] args)throws Exception {
Object resourceA = new Object();
Object resourceB = new Object();
Thread threadLockingResourceAFirst = new Thread(new DeadlockRunnable(resourceA, resourceB));
Thread threadLockingResourceBFirst = new Thread(new DeadlockRunnable(resourceA, resourceB));
threadLockingResourceAFirst.start();
Thread.sleep(500);
threadLockingResourceBFirst.start();
}
privatestaticclassDeadlockRunnableimplementsRunnable{
privatefinal Object firstResource;
privatefinal Object secondResource;
publicDeadlockRunnable(Object firstResource, Object secondResource){
this.firstResource = firstResource;
this.secondResource = secondResource;
}
@Overridepublicvoidrun(){
try {
synchronized (firstResource) {
printLockedResource(firstResource);
Thread.sleep(1000);
synchronized (secondResource) {
printLockedResource(secondResource);
}
}
} catch (InterruptedException e) {
System.out.println("Exception occurred: " + e);
}
}
privatestaticvoidprintLockedResource(Object resource){
System.out.println(Thread.currentThread().getName() + ": locked resource -> " + resource);
}
}
}This application will end without a deadlock, and as a result we will get the following output (note that the addresses of the Object class instances have changed):
Thread-0: locked resource -> java.lang.Object@1ad895d1 Thread-0: locked resource -> java.lang.Object@6e41d7dd Thread-1: locked resource -> java.lang.Object@1ad895d1 Thread-1: locked resource -> java.lang.Object@6e41d7dd
As a result, using only the information obtained from the thread dump, we were able to find the cause and fix the deadlock in our application. The proposed technique is sufficient for most simple applications (or applications with a small number of deadlocks). To deal with more complex situations, will require other methods.
Analysis of more complex thread dumps
Dumps of real applications can be very large and complex.
In one JVM hundreds of thousands of threads can be running at the same time. And more than two streams may be blocked (or there may be several threading problems caused by one problem).
The analysis of such a huge amount of information can be a real problem.
For analyzing large dumps, special analyzer utilities are used - Thread Dump Analyzers (TDAs). These utilities parse Java thread dumps and display information in a human-readable form, often using graphical tools. Moreover, some of them can perform static analysis and find the cause of the problem. Of course, the choice of a particular utility depends on a number of circumstances.
Nevertheless, here is a list of the most popular TDAs:
This is not a complete list of utilities of this kind. Each utility from this list performs a sufficiently qualitative analysis and simplifies the manual labor of analyzing dumps.
Conclusion
Thread dumps are a great tool for analyzing the state of a Java application, especially in cases of unexpected behavior of multi-threaded applications. However, without the proper baggage of knowledge, dumps can add additional complexity to an already difficult situation.
In this article, we developed a deadlock application, dumped the hung application. Analyzed the dump and found the cause of the lock and fixed it. This is not always so easy, for the analysis of most real-world applications it is very useful to use special utilities - analyzers.
However, every professional Java developer needs to understand the basics of thread dump analysis. Should be guided in their structure, know what information can be extracted and how to use it to solve problems of multithreading.
Although thread dump is not a “silver bullet” in the multithreading world, it is nevertheless an important tool for diagnosing complex, but quite common problems of multi-threaded Java applications.
In the Java Developer course program, multi-threading issues occupy a noticeable part. We consider in detail how to develop programs so that we don’t have to deal with deadlock production at night.
As always, your opinions and comments are interesting, which you can leave here or have a look at Vitaly on an open day .