About building JDK 8 on Ubuntu, the quality of Hotspot code, and why everything is felled in C ++
I wanted to sleep today, but again failed. There was a message in the Telegram that someone was not going to have Java ... and we woke up only after a couple of hours, tired and satisfied.
Who can this post be useful for? Yes, probably, to anyone except those who also collect JDK8 or just like to read horrible horrors. In general, I warned you, close the article urgently.
Three problems:
- Not going ( level one )
Very boring part that can be skipped. Only needed for those who want to fully restore the history of events; - It is not going to ( level two ) more
interesting, because there are a couple of common mistakes, necromancy, necrophilia, what BSD is better than GNU / Linux and why it is worth switching to new versions of JDK. - Even if going, falls into the crust
More interesting. Yahuuu, JVM fell into the crust, let's kick it with your feet!
Under the cut is shown a detailed course of solving problems, with different side thoughts about life.
There will be a lot of C ++, there will be no Java code at all. In the end, any javist starts writing only in C ++ ...
Not going
Anyone who has collected Java at least once knows that it looks something like this:
hg clone http://hg.openjdk.java.net/jdk8u/jdk8u
cd jdk8u
sh ./get_source.sh
sh ./configure \
--with-debug-level=fastdebug \
--with-target-bits=64 \
--with-native-debug-symbols=internal \
--with-boot-jdk=/home/me/opt/jdk1.8.0_161
make images(All my users are simply called “me”, so that the virtual user can at any time give to any person and not create a rejection from using it not by his username)
The problem, of course, is that it does not work. And quite cynical way.
First level of immersion
Let's try to run:
/home/me/git/jdk8u/hotspot/src/os/linux/vm/os_linux.inline.hpp:127:18: warning: ‘int readdir_r(DIR*, dirent*, dirent**)’ is deprecated [-Wdeprecated-declarations]
if((status = ::readdir_r(dirp, dbuf, &p)) != 0) {
^~~~~~~~~Initially, for you to understand, I have installed this:
$ g++ --version
g++ (Ubuntu 7.3.0-16ubuntu3) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.The compiler is not the first freshness, not 8.2, but this one should come up.
C ++ developers love to test software only on the version of the compiler that they have installed. Usually, the desire to test on different platforms ends somewhere around the difference between gcc and clang in a general sense. Therefore, it is quite normal to first vpendyurit -Werror(“consider warnings as errors”) and then write such code, which in all other versions will be considered vornings.
The problem is known, and it is clear how to solve it. You need to set your environment variable CXX_FLAGS, in which to prescribe the correct level of error.
export CXX_FLAGS=-Wno-error=deprecated-declarations -Wno-error-deprecated-declarationsAnd then we see the miraculous:
Ignoring CXXFLAGS foundin environment. Use --with-extra-cxxflagsOkay, build system, whatever you want! We substitute configure with this:
hg clone http://hg.openjdk.java.net/jdk8u/jdk8u
cd jdk8u
sh ./configure \
--with-extra-cflags='-Wno-cpp -Wno-error=deprecated-declarations' \
--with-extra-cxxflags='-Wno-cpp -Wno-error=deprecated-declarations' \
--with-debug-level=fastdebug \
--with-target-bits=64 \
--with-native-debug-symbols=internal \
--with-boot-jdk=/home/me/opt/jdk1.8.0_161
make imagesAnd the error remains the same!
Go to the heavy artillery: grepe sources.
grep -rl "Werror" .A huge amount of any auto-generated hat falls out, among which there are glimpses of meaningful files:
./common/autoconf/flags.m4
./hotspot/make/bsd/makefiles/gcc.make
./hotspot/make/solaris/makefiles/gcc.make
./hotspot/make/aix/makefiles/xlc.makeWe flags.m4easily find the previous message about “Ignoring CXXFLAGS” and a more hard-witted hard flag CCXX_FLGS(yes, two letters C), which immediately acts instead of CFLAGSand instead of СXX_FLAGS. Conveniently! Two facts are interesting:
- This flag is not passed through the configure parameters;
- The default values are meaningful and suspiciously similar to the real parameters:
# Setup compiler/platform specific flags to CFLAGS_JDK,# CXXFLAGS_JDK and CCXXFLAGS_JDK (common to C and CXX?)iftest"x$TOOLCHAIN_TYPE" = xgcc; then# these options are used for both C and C++ compiles
CCXXFLAGS_JDK="$CCXXFLAGS$CCXXFLAGS_JDK -Wall -Wno-parentheses -Wextra -Wno-unused -Wno-unused-parameter -Wformat=2 \
-pipe -D_GNU_SOURCE -D_REENTRANT -D_LARGEFILE64_SOURCE"This question looks very nice in the comments - is it a common flag? True?
Let's not play democracy and authoritarian zhidkodim there -w("do not show any errors"):
CCXXFLAGS_JDK="$CCXXFLAGS$CCXXFLAGS_JDK -w -ffreestanding -fno-builtin -Wno-parentheses -Wno-unused -Wno-unused-parameter -Wformat=2 \And - hooray! - we passed the first mistake. She is no longer reporting, and in general everything is fine. It would seem that.
Second level of immersion
But now it falls in a bunch of other new places!
It turns out that ours -wworks, but it is not forwarded to all parts of the assembly. Carefully read the makefiles and don’t understand how exactly this parameter can even be forwarded. Have you forgotten about him?
Knowing the right question to google (“why cxx doesn't get to build ?!”), we quickly get to the bug page with the saying name “configure --with-extra-cxxflags doesn't affect hotspot” ( JDK-8156967 ).
Which promise to fix in JDK 12. Maybe. Wonderful - the most important build option is not used in the assembly!
The first idea - well, let's roll up our sleeves and correct mistakes!
Error 1. xn [12]
dependencies.cpp: Infunction ‘staticvoidDependencies::write_dependency_to(xmlStream*, Dependencies::DepType, GrowableArray<Dependencies::DepArgument>*, Klass*)’:
dependencies.cpp:498:6: error: ‘%d’ directivewritingbetween 1 and 10 bytesintoaregionofsize 9 [-Werror=format-overflow=]voidDependencies::write_dependency_to(xmlStream* xtty,
^~~~~~~~~~~~
dependencies.cpp:498:6: note: directiveargumentintherange[0, 2147483647]Well, we probably need to increase the region. One hundred pounds someone calculated the buffer by pressing the "I'm feeling lucky!" Button in Google.
But how to understand how much is necessary? Below is a clarification of another kind:
stdio2.h:34:43: note: ‘__builtin___sprintf_chk’ output between3and12 bytes into a destination of size 10
__bos (__s), __fmt, __va_arg_pack ());Position 12 looks like something worthwhile, with which you can now break your dirty feet into the source.
We climb in dependencies.cppand observe the following picture:
DepArgument arg = args->at(j);
if (j == 1) {
if (arg.is_oop()) {
xtty->object("x", arg.oop_value());
} else {
xtty->object("x", arg.metadata_value());
}
} else {
char xn[10]; sprintf(xn, "x%d", j);
if (arg.is_oop()) {
xtty->object(xn, arg.oop_value());
} else {
xtty->object(xn, arg.metadata_value());
}
}Pay attention to the problem line:
char xn[10]; sprintf(xn, "x%d", j);We change 10 for 12, we rebuild and ... the assembly has gone!
But am I really the only one so smart and fixed the bug of all times and peoples? Not a question, again we drive our megapatch into Google:char xn[12];
And we see ... yes, that's right. The JDK-8184309 bug , made by Vladimir Ivanov, contains exactly the same fix.
But the point is that it is fixed only in JDK 10 and it is not backported to jdk8u. This is the question of why we need new versions of Java.
Error 2. strcmp
fprofiler.cpp: In member function ‘void ThreadProfiler::vm_update(TickPosition)’:
/home/me/git/jdk8ut/hotspot/src/share/vm/runtime/fprofiler.cpp:638:56: error: argument 1nullwhere non-null expected [-Werror=nonnull]
bool vm_match(const char* name) const { return strcmp(name, _name) == 0; }Taught by previous bitter experience, we immediately go to see what is in this place in JDK 11. And ... this file is not there. The directory structure also underwent some refactoring.
But it’s not so easy to leave us!
Any javist - in his heart a little necromancer, and maybe even a necrophil. Therefore, now there will be NONCOMANTAINY IN ACTION!
First you need to appeal to the soul of the dead and find out when he died:
$ hg log--template "File(s) deleted in rev {rev}: {file_dels % '\n {file}'}\n\n" -r 'removes("**/fprofiler.cpp")'
File(s) deleted in rev 47106:
hotspot/src/share/vm/runtime/fprofiler.cpp
hotspot/src/share/vm/runtime/fprofiler.hpp
hotspot/test/runtime/MinimalVM/Xprof.javaNow you need to find out the cause of his death:
hg log -r 47106
changeset: 47106:bed18a111b90
parent: 47104:6bdc0c9c44af
user: gziemski
date: Thu Aug 3120:26:532017-0500summary: 8173715: Remove FlatProfiler
So, we have a killer: gziemski . Let's find out why he nailed this unfortunate file.
To do this, go through the fat in the ticket specified in the summary of the commit. This is JDK-8173715 :
Remove FlatProfiler:
scanning for the GC.
For-shi-be. In fact, we are now offered to repair the corpse just to build it. Which has decomposed so much that even our necromancer colleagues from OpenJDK have abandoned.
Let's revive the dead man and try to ask him what he remembered last. He was already dead in revision 47106, so in revision, one less is “a second before”:
hg cat "~/git/jdk11/hotspot/src/share/vm/runtime/fprofiler.cpp" -r 47105 > ~/tmp/fprofiler_new.cpp
cp ~/git/jdk8u/hotspot/src/share/vm/runtime/fprofiler.cpp ~/tmp/fprofiler_old.cpp
cd ~/tmp
diff fprofiler_old.cpp fprofiler_new.cppUnfortunately, there is absolutely nothing concerning return strcmp(name, _name) == 0;the differential. Potsienta died of a blow with a blunt sharp object (rm utility), but at the time of death he was incurably ill.
Let's dig into the essence of the error.
Here is what the author of the code would like to tell us:
constchar *name()const{ return _name; }
boolis_compiled()const{ returntrue; }
boolvm_match(constchar* name)const{ returnstrcmp(name, _name) == 0; }Now a little philosophy.
The C11 standard in clause 7.1.4, “Use of library functions”, explicitly says:
The following is a statement of value (such as [...] a null pointer [...]) [...] the behavior is undefined.
That is, now the whole question is whether there is some kind of “explicitly stated otherwise” . In the description strcmpin section 7.24.4, nothing like this is written, and I have no other sections for you.
That is, we have an undefined behavior here.
Of course, you can take and rewrite this piece of code, surrounding it with a check. But I'm not at all sure that I correctly understand the logic of people who use UB where it should not be. For example, some systems generate SIGSERV zero dereferencing, and a hacker lover can plan on it, but this behavior is not necessary and can bang on another platform.
Yes, of course, someone will say that you are your own fool, that you are using GCC 7.3, but in GCC 4 everything would be fine. But undefined behavior! = Unspecified! = Implementation defined. This for the last two can be laid to work in the old compiler. And UB in the sixth version was UB.
In short, I was very sad over this difficult philosophical question (whether to crawl with my assumptions into the code) when I suddenly realized that it was possible and different.
There is another way
As you know, good heroes always go around.
Even apart from our philosophy about UB, there are an incredible amount of problems there. Not the fact that they can be repaired until the morning. Not the fact that I’m not crooked with my crooked hands. Even less is the fact that it will be accepted upstream: the last patch in jdk8u was 6 weeks ago, and it was the global merge of the new tag.
Just imagine that the code above is actually written correctly. All that stands between us and its execution is a kind of warning that was perceived as an error due to a bug in the build system. But we can build the assembly system.
The Witcher Geralt of Rivia used to say:
“Evil is evil, Stregobor,” the Witcher said seriously, getting up. - The smaller, the larger, the middle — everything is one, the proportions are conditional, and the borders are blurred. I am not a holy hermit, not only one good did in life. But if you have to choose between one evil and another, I prefer not to choose at all.
- Zło to zło, Stregoborze - rzekł poważnie wiedźmin wstając. - Mniejsze, większe, średnie, wszystko jedno, proporcje są umowne a granice zatarte. Nie jestem świątobliwym pustelnikiem, nie samo dobro czyniłem w życiu. Ale jeżeli mam wybierać pomiędzy jednym złem a drugim, to wol nie wybierać wcale.
This is a quote from the book "Last Wish," the story "Lesser Evil." We know that Geralt almost never could play the role of a truly neutral character until the end, and even died due to regular classical chaotic good behavior.
So let's zashkvarimsya about the lesser evil. Loosen the assembly system.
At the very beginning, we already saw this exhaust:
grep -rl "Werror" .
./common/autoconf/flags.m4
./hotspot/make/linux/makefiles/gcc.make
./hotspot/make/bsd/makefiles/gcc.make
./hotspot/make/solaris/makefiles/gcc.make
./hotspot/make/aix/makefiles/xlc.makeComparing these two files, I broke the whole face with a facepalm and realized the difference in the culture of the two platforms:
BSD is a story about freedom and choices:
# Compiler warnings are treated as errors
ifneq ($(COMPILER_WARNINGS_FATAL),false)
WARNINGS_ARE_ERRORS = -Werror
endifGNU / Linux is an authoritarian purist mode:
# Compiler warnings are treated as errorsWARNINGS_ARE_ERRORS = -WerrorWell still it would be forwarded in linux through ССXX_FLAGS, this variable WARNINGS_ARE_ERRORSis not taken into account when calculating ! In the GNU / Linux build, we simply have no choice but to follow defaults lowered over.
Well, or you can make it easier and change the value WARNINGS_ARE_ERRORSfor a short, but no less powerful -w. How do you like that, Ilon Musk?
As you might have guessed, this completely solves this build problem.
When the code is collected, you see a bunch of strange, creepy looking problems flying by. Sometimes it was so scary that I really wanted to press ctrl + C and try to figure it out. But no, it is impossible, it is impossible ...
It seems that everything has gathered and has not brought any additional problems. Although I, of course, did not dare to start testing. Still, the night, the eyes begin to stick together, but somehow I don’t want to switch to the last resort - the four banks of energy from the refrigerator.
Falls into the crust
The build is over, the executables are generated, we are great.
And here we are at the finish line. Or did not come?
Our assembly is in the following way:
export JAVA_HOME=~/git/jdk8u/build/linux-x86_64-normal-server-fastdebug/jdk
export PATH=$JAVA_HOME/bin:$PATHWhen you try to run the executable file, javait instantly falls into the crust. For those who are not familiar - it looks like this:
At the same time, Alex has Debian 9.5, and I have Ubuntu. Two different versions of GCC, two different looking crusts. I have innocent pranks with a manual strcmp patch and a few more places, Alex doesn’t. What is the problem?
This story is worthy of a separate story, but here let's go straight to dry conclusions, otherwise I will never finish this post.
The problem is that our favorite C ++ - pogromists again used undefined behavior.
(And where it is in an unknown way depends on the compiler implementation. However, we must remember that UB is always UB, even on a known version of the compiler, it cannot be laid on it)
In one place, we turn to the field of an undesigned class there, and everything breaks down. Do not ask how it happened, everything is difficult.
It is very difficult for a javista to imagine how you can refer to an undesigned class, except by releasing a link to it directly from the constructor. Fortunately, the wonderful C ++ language can do everything or almost everything. I will write an example with some kind of pseudocode:
classA
{
A()
{
_b.Show();
}
private:
static B _b;
};
A a;
B A::_b;
intmain(){
}Have a nice debug!
If you look at C ++ 98 [class.cdtor]:
For an object of non-POD class ...
Starting with GCC of some version (and I have 7.3), optimization of “lifetime dead store elimination” has appeared, which considers that we only turn to the object during its liftime, and mows it out of lifetime.
The solution is to disable the new optimizations and return them as they were in the old GCC:
CFLAGS += -fno-strict-aliasing -fno-lifetime-dse -fno-delete-null-pointer-checksThere is a discussion about this here .
For some reason, the panelists decided that this would not be included in the upstream. But you still need to try to send.
Add these options to ours ./hotspot/make/linux/makefiles/gcc.make, rebuild everything again and see the cherished lines:
t$ ~/git/jdk8u/build/linux-x86_64-normal-server-fastdebug/jdk/bin/java -version
openjdk version "1.8.0-internal-fastdebug"
OpenJDK Runtime Environment (build 1.8.0-internal-fastdebug-me_2018_09_10_08_14-b00)
OpenJDK 64-BitServer VM (build 25.71-b00-fastdebug, mixed mode)Conclusion
You probably thought that the output would be the following: “Java is some kind of hell, there is no garbage in the code, there is no support, everything is bad.”
This is not true! On the contrary, the examples above show from what terrible evil our friends, necromancers from OpenJDK, keep us.
And despite the fact that they have to live and use C ++, tremble from each UB and change the compiler version and learn the details of the platforms, the final user code in Java is insanely stable, and on the builds posted on the official websites of companies such as Azul, Red Hat and Oracle, it is hardly possible to run into a crust in the simple case.
The only sad thing is that most likely, the errors found are unlikely to be accepted in jdk8u. We took JDK 8 simply because it is easier for us to patch it right here and now, and we will have to deal with JDK 11. Nevertheless, to use JDK 8 in 2018 - IMHO, this is a very bad practice, and we are not doing this from a good life. Perhaps in the future our life will improve, and you will read many more incredible stories from the world of JDK 11 and JDK 12.
Thank you for the attention paid to such a boring text without pictures :-)
Minute advertising. The Joker 2018 conference will take place very soon, featuring many prominent Java and JVM specialists. View the full list of speakers and reports on the official website . I'll be there too, it will be possible to meet and grind for life and OpenJDK.