How to build your JDK, without blackjack and automatic garbage collection

gvsmirnov April 23, 2012 at 07:37

How to build your JDK, without blackjack and automatic garbage collection

Tutorial

At a recent Java One, Ruslan cheremin talked about how Disruptor developers use the JVM without a garbage collector. They had their own reasons for that, which have nothing to do with this topic.

I have long wanted to dig into the source code of a virtual machine, and cutting GC out of it is a great start. Under the cut, I will tell you how to assemble OpenJDK, cut out the garbage collector from it, and collect again. Towards the end, an answer will even be given to the question “why” that has come to your mind.

Sources? Give two more and sprinkle with binaries!

Main course

OpenJDK is stored in mercurial using forest, and the easiest way to get the code is to say

$ hg fclone http://hg.openjdk.java.net/jdk7/jdk7

If the forest extension is not installed and you do not want to install it for some reason, you can do this:

$ hg clone http://hg.openjdk.java.net/jdk7/jdk7 && jdk7/get_source.sh

Another option is to download full bundles from offsite. This will help to cut a couple of corners, but will deprive the delights of using a version control system.

An interesting feature: for some reason, jaxp and jaxws are stored in a separate repository. Therefore, they must either be manually downloaded from the appropriate sites ( jaxp.java.net and jax-ws.java.net ), or simply allowed to makedownload everything you need yourself by saying ALLOW_DOWNLOADS=true. Personally, this option seems more convenient to me. Oh yes, in the full source bundles everything has already been downloaded for us.

Tools without which the dish cannot be cooked

Clear business, assembly will require a lot of everything. The simplest is bootstrap jdk, at least version 1.6. It is necessary to indicate the path to it through a variable ALT_BOOTDIR. In addition, a huge pile of everything is required, from the obvious antand make to CUPS and ALSA . The easiest way to have exactly everything is to ask your package manager to satisfy all build dependencies. For example, using aptitude:

$ aptitude build-dep openjdk-6

Check what is going to

In order to make sure that everything you need is there, you need to run makewith a goal sanity. Pay attention to setting environment variables:

$ LANG=C ALT_BOOTDIR=/usr/lib/jvm/java-6-openjdk make sanity

If everything is good, then you will see the inscription If everything is bad, then you will receive a rather intelligible error message. Correct it and try again. Now you can build jdk itself. The environment variables added above .Sanity check passed

ALLOW_DOWNLOADS

$ ALLOW_DOWNLOADS=true LANG=C ALT_BOOTDIR=/usr/lib/jvm/java-6-openjdk make

In case of success in 20-40 minutes you will receive a message of the form

#-- Build times ----------
Target all_product_build
Start 2012-04-20 01:56:53
End   2012-04-20 02:02:14
00:00:06 corba
00:00:09 hotspot
00:00:06 jaxp
00:00:08 jaxws
00:04:47 jdk
00:00:05 langtools
00:05:21 TOTAL

You can verify that something useful has actually gathered and go to the next step.

$ ./build/linux-amd64/bin/java -version
openjdk version "1.7.0-vasily_p00pkin"
OpenJDK Runtime Environment (build 1.7.0-vasily_p00pkin-gs_2012_04_20_01_06-b00)
OpenJDK 64-Bit Server VM (build 23.0-b21, mixed mode)

I have an alternative operating system ...

... Based on BSD

It's not so bad here. Under the strict guidance of good Oracle employees, I managed to assemble a hotspot on a macbook in the Sapsan's vestibule. But the whole JDK the next night wasn’t very good. However, you can do this, you only need to have fresh Xcode and a lot of patience. I didn’t have either one or the other, and therefore I just started a more powerful machine in the Selectel cloud and conducted experiments on it. As a bonus, the assembly in the cloud is faster, while not loading my laptop, and therefore I can do something useful at this time (instead of fighting with swords, riding on chairs). If you still want to collect on the poppy, then there is a description of the process.

... Well, you get it, right?

Here, in fact, not everything is so bad either. Arm yourself with cygwin and smoke mana .

The beginning of the most interesting

- Patient, do you suffer from perversions?
- What are you, doctor! I enjoy them!
Now we are faced with the task of understanding where in the source you need to conjure to cut out the garbage collector. There are three obvious ways to do this: ask someone who knows, read all the source code, or show cunning and resourcefulness. The first method did not work out, because knowledgeable people looked at me strangely and moved away, refusing to participate in such dubious actions. The second method, although from the ideological point of view it was the most correct, was a pity for the time. Therefore, the third method remained.

Let us reason logically: how can someone influence the garbage collector from the outside? Two ways immediately come to mind: with the help of keys at startup (like -XX:+UseParallelGC) and withSystem.gc(). And although the former seems more logical, I decided to start from the latter, because javadocs cannot fully satisfy the interest as to what exactly is happening there. In java sources, this call is delegated to Runtime, where the method is already native. Everyone who has ever worked with the JNI, know how to write the names of functions in native code: Java_java_lang_Runtime_gc. Quick grepprompts such code in jdk/src/share/native/java/lang/Runtime.c, in which we are interested in the following lines:

JNIEXPORT void JNICALL
Java_java_lang_Runtime_gc(JNIEnv *env, jobject this)
{
    JVM_GC();
}

Clearly, now we are looking JVM_GC. No less quickly we find his ad in src/share/vm/prims/jvm.cpp:


JVM_ENTRY_NO_ENV(void, JVM_GC(void))
   JVMWrapper("JVM_GC");
   if (!DisableExplicitGC) {
     Universe::heap()->collect(GCCause::_java_lang_system_gc);
   }
JVM_END

Here we see already two very interesting points: the first - DisableExplicitGCwhich does not need comments and the collecty method Universe::heap(). How simple it is: it turns out that it System.gc()only does what the collector starts synchronously. No drama. Eh. Well, nothing, but now we know that, most likely, in the method collect()you can disable the assembly. We easily discover the Universe class in a file hotspot/src/share/vm/memory/universe.hppand notice that the static method heapreturns CollectedHeap*, as well as the presence of the methodinitialize_heap()

A little lyrical digression on the theme of the universe

I must say that the quality of the code in OpenJDK is excellent: a good structure, it is easy to understand what is happening, a lot of comments. For example, a great snippet:
121
122
123
124
125
126
127
class Universe: AllStatic {
  // Ugh.  Universe is much too friendly.
  friend class MarkSweep;
  friend class oopDesc;
  // Ещё куча friend'ов
  //...
}

Okay, back to our collector. The method initialize_heap()creates a heap, and depending on which collector the user specified, some specific implementation is selected. A complete list can be found in the file hotspot/src/share/vm/gc_interface/collectedHeap.hpp:


enum Name {
  Abstract,
  SharedHeap,
  GenCollectedHeap,
  ParallelScavengeHeap,
  G1CollectedHeap
};

Continuing the study of the class, finally we come across the necessary code:


// Perform a collection of the heap; intended for use in implementing
// "System.gc".  This probably implies as full a collection as the
// "CollectedHeap" supports.
virtual void collect(GCCause::Cause cause) = 0;
// This interface assumes that it's being called by the
// vm thread. It collects the heap assuming that the
// heap lock is already held and that we are executing in
// the context of the vm thread.
virtual void collect_as_vm_thread(GCCause::Cause cause) = 0;

Here the comments are most useful to us. For those who do not know English well enough, I’ll explain: the first method is simply collect()intended for assembly from the outside (for example, from System.gcor, as the same shows grep, with unsuccessful memory allocation in linux). The second is launched from the virtual machine thread, which is responsible for garbage collection (and it is assumed that all the necessary locks are already held). A simple solution immediately comes to mind: to make sure that when these methods are called, the assembly does not occur. I even tried this approach for the first time, but it’s bad luck: it turns out that everything is somewhat more complicated, and each implementation of the heap has its own additional places in which the assembly takes place. Therefore, I had to choose some specific implementation ( GenCollectedHeapwithMarkSweepPolicyas the simplest one), and depending on the flag (which I called UseTheForce), it can exit the methods that produce the assembly without doing anything. As a result, the changes in the first version happened like this .

We try!

We’ll quickly sketch a class that, during normal operation of the garbage collector, should not throw OOM, but if it is absent, it does so with great joy:

public class TheForceTester {
    public static final int ARRAY_SIZE = 1000000;
    public static void main(String[] args) {
        while (true) {
            byte[] lotsOfUsefulData = new byte[ARRAY_SIZE];
        }
    }
}

And start this business using our new virtual machine:

$ ./build/linux-amd64/bin/java -XX:+UseTheForce -verbose:gc TheForceTester
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at ru.yandex.holocron.core.TheForceTester.main(TheForceTester.java:10)

Hurrah! Fierce wines! Moreover, in the tester application, you can add the output of the current free space and make sure that everything else also seems to work correctly: the heap Xmx != Xmsexpands when it is equal, and if equal, the free space decreases exactly as much as it should in theory. Class! It remains only to add a spoonful of tar.

Disclaimer and yet the answer to the same Question

By the same Question, I, of course, mean "And Why ?!". At the beginning of the topic, I mentioned Disruptor, for which performance is extremely critical. The garbage collector, as you know, introduces poorly predictable delays in the application. Therefore, if it is possible to reuse most of the objects and restart from time to time, I drank GC - it’s quite an adequate way to speed up.

In addition, because I want to see if I can. Also curious.

Disclaimer is the following: the solution given is rather dirty, and serves more as a proof of concept. First of all, because we actually made garbage collection instant, leaving the various other overheads from using the collector in the virtual machine. In a good way, it was worth writing your implementationCollectedHeap, which all these overheads would have completely ruled out. However, even after that there would probably be a few more places where you would have to pick your own.

What follows from all this? Wait for more topics! :)

PS What else would you do?

PPS Archive compiled under linux-amd64: clck.ru/1-L-9 (Yandex.Disk)

PPPS Please do not clone my entire repository. It weighs 600+ megabytes, and traffic on the machine where it is hosted is paid. However, this does not stop you from leaning towards java.net, and then pulling a single commit ( 3358: 3f014511ecce ).

Tags:

How to build your JDK, without blackjack and automatic garbage collection