PermGen memory leak analysis in Java
What are you talking about?
Those who were involved in web development in Java, probably faced with such a problem as
java.lang.OutOfMemoryError: PermGen space. It occurs, as a rule, after restarting the web application inside the server without restarting the server itself. Restarting the web application without restarting the server may be necessary during the development process, so as not to wait for extra time to start the server itself. If you have multiple web applications deployed, restarting the entire server can be much longer than restarting one web application. Or the whole server simply cannot be restarted, as other web applications are used. The first solution that comes to mind is to increase the maximum amount of PermGen memory available to the JVM (this can be done as an option -XX:MaxPermSize), but this will only delay the crash, after several restarts you will again getOutOfMemoryError. It would be nice to be able to restart and redo the web application on a running server as many times as you like. About how to overcome PermGen, and further discussion will go.What is PermGen?
PermGen - Permanent Generation is a memory area in the JVM designed to store Java class descriptions and some additional data. Thus, when you restart the web application, all classes are loaded on a new one and fill PermGen memory. A web application can contain a bunch of libraries, and class descriptions can take tens of megabytes. Those who follow innovations in Java may have heard that PermGen has been abandoned in Java 8. Here you might think that the eternal problem was finally fixed, and there would no longer be crashes from the lack of PermGen memory. Unfortunately this is not the case, roughly speaking, PermGen is now simply called Metaspace, and you will still get it
OutOfMemoryError.Stop. But what about the garbage collector?
We all know that Java has a garbage collector that collects all unused objects. It should also collect unused classes in PermGen, but only if it is correctly configured and there are no memory leaks.
With regard to the setting - the official documentation is quite small, on the Internet there are plenty of tips to use different option, for example
-XX:+CMSClassUnloadingEnabled, -XX:+CMSPermGenSweepingEnabled, -XX:+UseConcMarkSweepGC. I did not dig deep and look for official documentation, but determined by trial and error that for Java 7 and Tomcat 7 it was necessary and sufficient to add the JVM option-XX:+UseConcMarkSweepGC. This option will change the garbage collection algorithm, if you are not sure that your application will not work worse because of this, then look for documentation and comparisons of the operation of different garbage collection algorithms to determine whether to use this option or not. Perhaps you just need to enable this option to get rid of PermGen problems. If not - then you most likely have a memory leak, what to do with it - read on.Why is a PermGen memory leak occurring?
To start, a few words about class loaders. Class loaders are the objects in Java that are responsible for loading classes. There is a hierarchy of class loaders in web servers, for each web application there is one class loader, plus several common class loaders. Classes within the web application are loaded by the class loader that corresponds to this web application. The system classes and classes required by the server itself are loaded by common class loaders. For example, how the hierarchy of class loaders for Tomcat is arranged, you can read here .
In order for the garbage collector to collect all classes of the web application, they should not be referenced outside this web application. Now remember that each object in Java stores a reference to its class, i.e. per class object
java.lang.Class, and each class stores a reference to the class loader that loaded this class, and each class loader stores links to all the classes that it loaded. It turns out that just one external link to the web application object pulls all the web application classes along with the inability to collect them with the garbage collector. 
Another cause of the leak could be a thread that was launched from a web application and that could not be stopped when the web application stopped. It also stores a link to a class loader web application.
Also a popular leak option is
ThreadLocalThe variable to which the object from the web application is assigned for the stream from the shared pool. In this case, the stream stores a reference to the object. A thread from the shared pool cannot be destroyed, that means the object cannot be destroyed, so the whole class loader with all classes cannot be destroyed.Standard Tomcat Tools
Fortunately, there are a number of tools in Tomcat to analyze and prevent PermGen memory leaks.
Firstly , the standard Tomcat Manager Application has a “Find leaks” button ( details ), which will analyze which web applications left trash after a restart.
But this will only show which web applications may contain a leak, there is little use for this.
Secondly , in Tomcat there
JreMemoryLeakPreventionListeneris a solution for the well-known possible options for memory leaks, it is configured in server.xml( details ). Perhaps including any options for this listener will help get rid of memory leaks. And thirdlythe most important thing is that when the web application is stopped, Tomcat writes to the log what exactly could lead to a memory leak. For example, like this:
SEVERE: The web application [/drp] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak.SEVERE: The web application [/drp] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@7dc1e95f]) and a value of type [java.util.Hashtable] (value [{session=*2CBFB7}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.This is just what we need to continue the leak analysis.
And since we seriously got down to business, we need to know how to properly check whether PermGen is cleared with us or not. Tomcat Manager Application, which can show memory usage, including PermGen, will help us with this again.
Another feature - cleaning occurs only after reaching the maximum PermGen memory volume, so you need to set a small value of the maximum available PermGen memory (for example, like this
-XX:MaxPermSize=100M), so that after two or three restarts of the web application, the occupied memory reaches 100%, and either the cleaning occurs , or fell OutOfMemoryErrorif the leaks still remained.Now let's look at how to get rid of leaks using examples.
Take the following message:
SEVERE: The web application [/drp] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak.It tells us that the web application started and did not stop the thread
AWT-Windows, therefore, it contextClassLoaderturned out to be the class loader of the web application, and the garbage collector cannot collect it. Here we can track using breakpoint with a condition by the name of the thread who created this thread and, digging through the source, find what possibilities it can be stopped, for example, put some flag or call some method, for example Thread#interrupt(). These steps will need to be performed when the web application is stopped. But you can also notice that the name of the stream seems to be something systemic ... Maybe
JreMemoryLeakPreventionListener, about which we learned above, something can do with this stream? We go into the documentation and see that the listener really has a parameterAWTThreadProtectionwhich for some reason is false by default. We put it in true in server.xml and make sure that Tomcat does not return such a message. In this case, the stream
AWT-Windowswas created due to the generation of captcha on the server using the classes for working with images from the JDK. Ok, here we got off with a simple option in Tomcat, let's try something more complicated:
SEVERE: The web application [/drp] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@7dc1e95f]) and a value of type [java.util.Hashtable] (value [{session=*2CBFB7}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.Here we see that someone put some value in the
ThreadLocalclass variable ThreadLocalMapand did not remove it. We look for where the class is used ThreadLocalMap, we find org.apache.log4j.MDC, and this class is already directly used in our web application to log additional information. We see that the putclass MDCmethod is removebeing called , but the method is not being called. It seems like a challenge removefor everyoneputin the right place should help. We fix, check - it works! After correcting all such errors, it is likely that you will get rid of
OutOfMemoryError: PermGen space, at least in my practice, this was so.VisualVM Analysis
If you are not using Tomcat, or if fixing the errors indicated by Tomcat in the log did not help, then you can continue the analysis using the profiler. I took the free VisualVM profiler included with the JDK.
First, start the server with one deployed web application and restart it so that the leak is visible. Open VisualVM, select the desired process and make a heap dump by selecting the appropriate item in the drop-down menu.
Select the “OQL Console” tab and execute the following query:
select x from org.apache.catalina.loader.WebappClassLoader x(for other servlet implementations the class will be different).
One of the two instances was left from the first stopped web application, the garbage collector could not collect it. To determine which one is the old one, click on one of them and look for the field
started. The old one startedwill false.
In the "References" window, all links to this class loader are shown, we need to find the one due to which the garbage collector cannot collect it. To do this, right-click on it
thisand select “Show Nearest GC Root”. 
Great, we found some thread that has our old class loader as the
contextClassloaderith. We click on it with the right mouse button and select “Show Instance”. 
We look at the fields of the object and think about what we can catch on to understand what kind of object it is, somehow find the code that creates it, catch it in the debugger, etc. In this case, this stream name is familiar to us
AWT-Windows. We found the same problem Tomcat wrote about to us, only with VisualVM. You already know how to solve it.Total
We have learned to identify, analyze and correct PermGen memory leaks. It turned out to be not so difficult, especially thanks to the built-in tools of Tomcat. I can not guarantee that the above methods can get rid of all types of leaks, however, I managed to get rid of leaks in several large projects in this way.