voyachek June 6, 2019 at 08:54

Java Native Image: Usability Check

Not so long ago, Oracle released the first release of the GraalVM project (https://www.graalvm.org/). The release was immediately assigned the number 19.0.0, apparently in order to convince that the project is mature and ready for use in serious applications. One of the parts of this project: Substrate VM is a framework that allows you to turn Java applications into native executable files (as well as native libraries that can be connected in applications written, for example, in C / C ++). This feature has so far been declared experimental. It is also worth noting that native Java applications have some limitations: you must list all the resources used to include them in the native program; you need to list all the classes that will be used with reflection and other restrictions. The full list is listed here.Native Image Java Limitations . Having studied this list, it’s understandable in principle that the restrictions are not so significant that it would be impossible to develop more complex applications than hellwords. I set this goal: developing a small program that has a built-in web server, uses a database (via the ORM library) and compiles into a native binary that can run on systems without a Java machine installed.

I will experiment on Ubuntu 19.04 (Intel Core i3-6100 CPU @ 3.70GHz × 4).

Install GraalVM

GraalVM installation is conveniently done using SDKMAN . GraalVM installation command:

sdk install java 19.0.0-grl

OpenJDK GraalVM CE 19.0.0 will be installed, CE is Community Edition. There is also Enterprise Edition (EE), but this edition needs to be downloaded from the Oracle Technology Network, the link is on the GraalVM Downloads page .

After installing GraalVM, already using the gu component update manager from GraalVM, I installed compilation support in the native binary -

gu install native-image

Everything, working tools are ready, now you can start developing applications.

Simple native application

As a build system, I use Maven. To create native binaries, there is a maven plugin:

native-image-maven-plugin

com.oracle.substratevmnative-image-maven-plugin${graal.version}native-imagepackagenativej
                    --no-server

It is still required to set the main class of the application. This can be done both in native-image-maven-plugin, and in the traditional way, via:

maven-jar-plugin

org.apache.maven.pluginsmaven-jar-plugin2.4nativej.Startup

Create the main class:

Startup.java

public class Startup {
    public static void main(String[] args) {
        System.out.println("Hello world!");
    }
}

Now you can run the maven command to build the application:

mvn clean package

Building a native binary on my machine takes 35 seconds. As a result, a binary file of 2.5 MB in size is obtained in the target directory. The program does not require the installed Java machine and runs on machines where Java is missing.

Repository link: Github: native-java-helloworld-demo .

JDBC Postgres Driver

And so, a simple application works, displays “Hello world”. No solutions were needed. I'll try to go one level higher: I will connect the Postgres JDBC driver to request data from the database. The Issues on githabe GraalVM come across bugs related to the Postgres driver, but release candidates GraalVM. All of them are marked as fixed.

I connect the postgresql dependency:

org.postgresqlpostgresql42.2.5

I am writing the code for extracting data from the database (the simplest user plate was created):

Startup.java

public class Startup {
    public static void main(String[] args) SQLException {
        final PGSimpleDataSource ds = new PGSimpleDataSource();
        ds.setUrl("jdbc:postgresql://localhost/demo_nativem");
        ds.setUser("test");
        ds.setPassword("test");
        try (
                Connection conn = ds.getConnection();
                Statement stmt = conn.createStatement();
                ResultSet rs = stmt.executeQuery("SELECT * FROM \"public\".\"user\"");
        ) {
            while(rs.next()){
                System.out.print("ID: " + rs.getLong("id"));
                System.out.println(", Name: " + rs.getString("name"));
            }
        }
    }
}

I collect the native binary and immediately get the build error: The fact is that the native application builder initializes all static fields during the build process (unless otherwise specified), and he does this by examining the dependencies of the classes. My code does not refer to org.postgresql.Driver, so the collector does not know how to initialize it better (when building, or when the application starts) and offers to register it for initialization when building. This can be done by adding it to the maven arguments of the native-image-maven-plugin plugin, as indicated in the error description. After adding Driver, I get the same error related to org.postgresql.util.SharedTimer. Again, I collect and encounter such a build error:

Error: No instances are allowed in the image heap for a class that is initialized or reinitialized at image runtime: org.postgresql.Driver. Try marking this class for build-time initialization with --initialize-at-build-time=org.postgresql.Driver

Error: Class initialization failed: org.postgresql.sspi.SSPIClient

There are no recommendations for correction. But, looking at the source of the class, it is clear that it relates to the execution of code under Windows. On Linux, its initialization (which occurs during assembly) fails with an error. There is an opportunity to delay its initialization at the start of the application: --initialize-at-run-time = org.postgresql.sspi.SSPIClient. Initialization on Linux will not occur and we will no longer get errors related to this class. Build Arguments:


    --no-server
    --no-fallback
    --initialize-at-build-time=org.postgresql.Driver
    --initialize-at-build-time=org.postgresql.util.SharedTimer
    --initialize-at-run-time=org.postgresql.sspi.SSPIClient

The assembly began to take 1 minute 20 seconds and the file swelled up to 11 MB. I added an additional flag for building the binary: --no-fallback forbids generating a native binary that requires an installed Java machine. Such a binary is created if the collector detects the use of language features that are either not supported in the Substrate VM or require configuration, but there is no configuration yet. In my case, the collector discovered the potential use of reflection in the JDBC driver. But this is only potential use, it is not required in my program, and therefore, additional configuration is not required (how to do it will be shown later). There is also the --static flag, which forces the generator to statically link libc. But if you use it, the program crashes with a segmentation fault when you try to resolve the network name to an IP address.

I run the resulting binary and get the following error: After some research, the cause of the error was identified: Postgres by default establishes a TLS connection using Elliptic Curve. SubstrateVM does not include the implementation of such algorithms for TLS, here is the corresponding open issue - Single-binary ECC (ECDSA / ECDHE) TLS support for SubstrateVM . There are several solutions: put the library from the GraalVM package: libsunec.so next to the application, configure the list of algorithms on the Postgres server, exclude Elliptic Curve algorithms or simply disable the TLS connection in the Postgres driver (this option was chosen):

Exception in thread "main" org.postgresql.util.PSQLException: Could not find a java cryptographic algorithm: TLS SSLContext not available.

dataSource.setSslMode(SslMode.DISABLE.value);

Having eliminated the error of creating a connection with Postgres, I launch the native application, it runs and displays data from the database.

Repository link: Github: native-java-postgres-demo .

DI framework and embedded web server

When developing a complex Java application, they usually use some kind of framework, for example, Spring Boot. But judging by this article by GraalVM native image support , the work of Spring Boot in the native image “out of the box” is promised to us only in Spring Boot 5.3.

But there is a wonderful Micronaut framework that claims to work in GraalVM native image. In general, connecting a Micronaut to an application that will be assembled in a binary does not require any special settings or problem solving. Indeed, many settings for using reflection and connecting resources for the Substrate VM are already made inside Micronaut. By the way, the same settings can be placed inside your application in the settings file META-INF / native-image / $ {groupId} / $ {artifactId} /native-image.properties (such a path for the settings file is recommended by Substrate VM), here’s a typical file content:

native-image.properties

Args = \
  -H:+ReportUnsupportedElementsAtRuntime \
  -H:ResourceConfigurationResources=${.}/resource-config.json \
  -H:ReflectionConfigurationResources=${.}/reflect-config.json \
  -H:DynamicProxyConfigurationResources=${.}/proxy-config.json \
  --initialize-at-build-time=org.postgresql.Driver \
  --initialize-at-build-time=org.postgresql.util.SharedTimer \
  --initialize-at-run-time=org.postgresql.sspi.SSPIClient

The files resource-config.json, reflect-config.json, proxy-config.json contain settings for connecting resources, reflection and proxies used (Proxy.newProxyInstance). These files can be created manually or retrieved using agentlib: native-image-agent. In the case of using native-image-agent, you need to run the usual jar (and not the native binary) using the agent:


java -agentlib:native-image-agent=config-output-dir=output -jar my.jar

where output is the directory where the files described above will be located. In this case, the program needs not only to run, but also to execute scripts in the program, because settings are written to the files as you use reflection, open resources, create a proxy. These files can be placed META-INF / native-image / $ {groupId} / $ {artifactId} and referenced in native-image.properties.

I decided to connect logging using logback: I added a dependency to the logback-classic library and the logback.xml file. After that, I compiled a regular jar and ran it using native-image-agent. At the end of the program, the necessary settings files. If you look at their contents, you can see that the agent registered the use of logback.xml to compile into the binary. Also, the reflection-config.json file contains all the cases of using reflection: for given classes, meta information will get into the binary.

Then I added a dependency to the micronaut-http-server-netty library to use the netty-based embedded web server and created a controller:

Startup.java

@Controller("/hello")
public class HelloController {
    @Get("/{name}")
    @Produces(MediaType.TEXT_PLAIN)
    public HttpResponse hello(String name) {
        return HttpResponse.ok("Hello " + name);
    }
}

And main class:

HelloController.java

public class Startup {
    public static void main(String[] args) {
        Signal.handle(new Signal("INT"), sig -> System.exit(0));
        Micronaut.run(Startup.class, args);
    }
}

Now you can try to build a native binary. My assembly took 4 minutes. If you run it and go to the address http: // localhost: 8080 / hello / user then an error occurs:

{"_links":{"self":{"href":"/hello/user","templated":false}},"message":"More than 1 route matched the incoming request. The following routes matched /hello/user: GET - /hello/user, GET - /hello/user"}

Honestly, it’s not entirely clear why this happens, but after searching by typing, I found that the error disappears if the following lines are removed from the resource-config.json file (which was created by the agent):

    {"pattern":"META-INF/services/com.fasterxml.jackson.databind.Module"}, 
    {"pattern":"META-INF/services/io.micronaut.context.env.PropertySourceLoader"}, 
    {"pattern":"META-INF/services/io.micronaut.http.HttpResponseFactory"}, 
    {"pattern":"META-INF/services/io.micronaut.inject.BeanConfiguration"}, 
    {"pattern":"META-INF/services/io.micronaut.inject.BeanDefinitionReference"},

Micronaut registers these resources and it seems that re-registration leads to double registration of my controller and an error. If after correcting the file, you rebuild the binary and run it, there will be no more errors, the text “Hello user” will be displayed at http: // localhost: 8080 / hello / user .

I want to draw attention to the use of the following line in the main class:

Signal.handle(new Signal("INT"), sig -> System.exit(0));

It needs to be inserted for Micronaut to finish correctly. Despite the fact that Micronaut hangs a hook to shut down, it does not work in the native binary. There is a corresponding issue: Shutdownhook not firing with native . It is marked as fixed, but, in fact, it has only a workaround using the Signal class.

Repository link: Github: native-java-postgres-micronaut-demo .

ORM connection

JDBC is good, but tires of repetitive code, endless SELECT and UPDATE. I’ll try to facilitate (or complicate, depending on which side to look) my life by connecting some kind of ORM.

Hibernate

At first I decided to try Hibernate , as it is one of the most common ORMs for Java. But I did not manage to build a native image using Hibernate due to a build error:

Error: Field java.lang.reflect.Method.defaultValue is not present on type java.lang.reflect.Constructor. Error encountered while analysing java.lang.reflect.Method.getDefaultValue() 
Parsing context:
	parsing org.hibernate.annotations.common.annotationfactory.AnnotationProxy.getAnnotationValues(AnnotationProxy.java:63)
	parsing org.hibernate.annotations.common.annotationfactory.AnnotationProxy(AnnotationProxy.java:52)
...

There is a corresponding open issue: [native-image] Micronaut + Hibernate results in Error encountered while analysing java.lang.reflect.Method.getDefaultValue () .

jOOQ

Then I decided to try jOOQ . I managed to build a native binary, though I had to make a lot of settings: specifying which classes to initialize (buildtime, runtime) and mess with reflection. In the end, it all came down to the fact that when the application starts, jOOQ initializes the proxy org.jooq.impl.ParserImpl $ Ignore as a static member of the class org.jooq.impl.Tools. And this proxy uses MethodHandle, which Substrate VM does not yet support . Here's a similar open issue: [native-image] Micronaut + Kafka fails to build native image with MethodHandle argument could not be reduced to at most a single call .

Apache cayenne

Apache Cayenne is less common, but looks quite functional. I'll try to connect it. I created XML files for describing the database schema, they can be created either manually or using the CayenneModeler GUI tool, or based on an existing database. Using the cayenne-maven-plugin in the pom file, code generation of classes that correspond to the database tables will be carried out:

cayenne-maven-plugin

org.apache.cayenne.pluginscayenne-maven-plugin${cayenne.version}src/main/resources/db/datamap.map.xml${project.build.directory}/generated-sources/cayennecgen

Then I added the CayenneRuntimeFactory class to initialize the database context factory:

CayenneRuntimeFactory.java

@Factory
public class CayenneRuntimeFactory {
    private final DataSource dataSource;
    public CayenneRuntimeFactory(DataSource dataSource) {
        this.dataSource = dataSource;
    }
    @Bean
    @Singleton
    public ServerRuntime cayenneRuntime() {
        return ServerRuntime.builder()
                .dataSource(dataSource)
                .addConfig("db/cayenne-test.xml")
                .build();
    }
}

HelloController Controller:

HelloController.java

@Controller("/hello")
public class HelloController {
    private final ServerRuntime cayenneRuntime;
    public HelloController(ServerRuntime cayenneRuntime) {
        this.cayenneRuntime = cayenneRuntime;
    }
    @Get("/{name}")
    @Produces(MediaType.TEXT_PLAIN)
    public HttpResponse hello(String name) {
        final ObjectContext context = cayenneRuntime.newContext();
        final List result = ObjectSelect.query(User.class).select(context);
        if (result.size() > 0) {
            result.get(0).setName(name);
        }
        context.commitChanges();
        return HttpResponse.ok(result.stream()
                .map(x -> MessageFormat.format("{0}.{1}", x.getObjectId(), x.getName()))
                .collect(Collectors.joining(",")));
    }
}

Then he launched the program as a regular jar, using agentlib: native-image-agent, to collect information about the resources used and reflexion.

I collected the native binary, run it, go to the address http: // localhost: 8080 / hello / user and get an error:

{"message":"Internal Server Error: Provider com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl not found"}

it turns out agentlib: native-image-agent did not detect the use of this class in reflection.

Manually added it to the reflect-config.json file:

{
  "name":"com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl",
  "allDeclaredConstructors":true
}

Again, I collect the binary, start, update the web page and get another error:

Caused by: java.util.MissingResourceException: Resource bundle not found org.apache.cayenne.cayenne-strings. Register the resource bundle using the option -H:IncludeResourceBundles=org.apache.cayenne.cayenne-strings.

Everything is clear here, I add the setting, as indicated in the proposed solution. Again I collect the binary (this is 5 minutes of time), I start it again and again an error, another:

No DataMap found, can't route query org.apache.cayenne.query.SelectQuery@2af96966[root=class name.voyachek.demos.nativemcp.db.User,name=]"}

I had to tinker with this error, after numerous tests, studying the sources, it became clear that the reason for the error lies in this line from the class org.apache.cayenne.resource.URLResource:

return new URLResource(new URL(url, relativePath));

As it turned out, Substrate VM loads the resource by the url, which is indicated as the base, and not by the url, which should be formed on the basis of the base and relativePath. What the following issue was registered by me about: Invalid resource content when using new URL (URL context, String spec) .

The error is determined, now you need to look for workarounds. Fortunately, Apache Cayenne turned out to be quite a customizable thing. You had to register your own resource loader:

ServerRuntime.builder()
                .dataSource(dataSource)
                .addConfig("db/cayenne-test.xml")
                .addModule(binder -> {
                    binder.bind(ResourceLocator.class).to(ClassLoaderResourceLocatorFix.class);
                    binder.bind(Key.get(ResourceLocator.class, Constants.SERVER_RESOURCE_LOCATOR)).to(ClassLoaderResourceLocatorFix.class);
                })
                .build();

Here is his code:

ClassLoaderResourceLocatorFix.java

public class ClassLoaderResourceLocatorFix implements ResourceLocator {
    private ClassLoaderManager classLoaderManager;
    public ClassLoaderResourceLocatorFix(@Inject ClassLoaderManager classLoaderManager) {
        this.classLoaderManager = classLoaderManager;
    }
    @Override
    public Collection findResources(String name) {
        final Collection resources = new ArrayList<>(3);
        final Enumeration urls;
        try {
            urls = classLoaderManager.getClassLoader(name).getResources(name);
        } catch (IOException e) {
            throw new ConfigurationException("Error getting resources for ");
        }
        while (urls.hasMoreElements()) {
            resources.add(new URLResourceFix(urls.nextElement()));
        }
        return resources;
    }
    private class URLResourceFix extends URLResource {
        URLResourceFix(URL url) {
            super(url);
        }
        @Override
        public Resource getRelativeResource(String relativePath) {
            try {
                String url = getURL().toString();
                url = url.substring(0, url.lastIndexOf("/") + 1) + relativePath;
                return new URLResource(new URI(url).toURL());
            } catch (MalformedURLException | URISyntaxException e) {
                throw new CayenneRuntimeException(
                        "Error creating relative resource '%s' : '%s'",
                        e,
                        getURL(),
                        relativePath);
            }
        }
    }
}

It has a line

return new URLResource(new URL(url, relativePath));

replaced by:

String url = getURL().toString();
url = url.substring(0, url.lastIndexOf("/") + 1) + relativePath;
return new URLResource(new URI(url).toURL());

I collect the binary (70 MB), start it, go to http: // localhost: 8080 / hello / user and everything works, the data from the database is displayed on the page.

Repository link: Github: native-micronaut-cayenne-demo .

conclusions

The goal has been achieved: a simple web application with access to the database using ORM has been developed. The application is compiled into a native binary and can run on systems without a Java machine installed. Despite numerous problems, I found a combination of frameworks, settings, and workarounds that allowed me to get a working program.

Yes, the ability to build regular binaries from Java source code is still in experimental status. This is evident by the abundance of problems, the need to look for workarounds. But in the end, it still turned out to achieve the desired result. What did I get?

The only self-contained file (almost, there are dependencies on libraries such as libc) that can run on systems without a Java machine.
Start time is an average of 40 milliseconds versus 2 seconds when starting a regular jar.

Among the shortcomings, I would like to note the long compilation time of the native binary. It takes me an average of five minutes, and most likely it will increase when writing code and connecting libraries. Therefore, it makes sense to create binaries based on fully debugged code. In addition, debugging information for native binaries is available only in the commercial edition of Graal VM - Enterprise Edition.

Tags:

Java Native Image: Usability Check

Install GraalVM

Simple native application

JDBC Postgres Driver

DI framework and embedded web server

ORM connection

Hibernate

jOOQ

Apache cayenne

conclusions

Also popular now: