Parse lambda expressions in java

Original author: Evgeniy Kirichenko, Roman Sakno
  • Transfer

image


From the translator: LambdaMetafactory is perhaps one of the most underrated mechanisms of Java 8. We discovered it recently, but we already appreciated its capabilities. In version 7.0 of the CUBA framework , performance is improved by eliminating reflective calls in favor of generating lambda expressions. One of the applications of this mechanism in our framework is to bind application event handlers by annotations, a common task, an analogue of EventListener from Spring. We believe that knowledge of how LambdaFactory works can be useful in many Java applications, and we hasten to share this translation with you.


In this article, we will show some little-known tricks when working with lambda expressions in Java 8 and the limitations of these expressions. The target audience of the article is senior Java developers, researchers and toolkit developers. Only public Java API without com.sun.*other inner classes will be used , so the code is portable between different JVM implementations.


Short preface


Lambda expressions appeared in Java 8 as a way of implementing anonymous methods and,
in some cases, as an alternative to anonymous classes. At the bytecode level, the lambda expression is replaced by the instruction invokedynamic. This instruction is used to create the implementation of the functional interface and its only method delegates the call to the actual method that contains the code defined in the body of the lambda expression.


For example, we have the following code:


voidprintElements(List<String> strings){
    strings.forEach(item -> System.out.println("Item = %s", item));
}

This code will be converted by the Java compiler into something like: 


privatestaticvoidlambda_forEach(String item){ //сгенерировано Java компилятором
    System.out.println("Item = %s", item);
}
privatestatic CallSite bootstrapLambda(Lookup lookup, String name, MethodType type){ ////lookup = предоставляется VM//name = "lambda_forEach", предоставляется VM//type = String -> void
    MethodHandle lambdaImplementation = lookup.findStatic(lookup.lookupClass(), name, type);
    return LambdaMetafactory.metafactory(lookup,
        "accept",
        MethodType.methodType(Consumer.class), //сигнатура фабрики лямбда-выражений
        MethodType.methodType(void.class, Object.class), //сигнатура метода Consumer.accept после стирания типов  
        lambdaImplementation, //ссылка на метод с кодом лямбда-выражения
        type);
}
voidprintElements(List<String> strings){
    Consumer<String> lambda = invokedynamic# bootstrapLambda, #lambda_forEach
    strings.forEach(lambda);
}

The instruction invokedynamiccan be approximately represented as such Java code:


privatestatic CallSite cs;
voidprintElements(List<String> strings){
    Consumer<String> lambda;
    //begin invokedynamicif (cs == null)
        cs = bootstrapLambda(MethodHandles.lookup(), 
                            "lambda_forEach", 
                            MethodType.methodType(void.class, String.class));
    lambda = (Consumer<String>)cs.getTarget().invokeExact();
    //end invokedynamic
    strings.forEach(lambda);
}

As you can see, it LambdaMetafactoryis used to create a CallSite that provides a factory method that returns a target method handler. This method returns the implementation of the functional interface using invokeExact. If there are captured variables in the lambda expression, then it invokeExactaccepts these variables as actual parameters.


In Oracle JRE 8, the metafactory dynamically generates a Java class using ObjectWeb Asm, which creates the implementation class of the functional interface. Additional fields can be added to the created class if the lambda expression captures external variables. This is similar to anonymous Java classes, but there are the following differences:


  • An anonymous class generated by a Java compiler.
  • The class for implementing lambda expressions is created by the JVM at runtime.


The implementation of the metafactory depends on the JVM vendor and on the version



Of course, the instruction is invokedynamicused not only for lambda expressions in Java. It is mainly used when running dynamic languages ​​in a JVM environment. The Nashorn engine for JavaScript execution, which is embedded in Java, makes heavy use of this instruction.


Next, we focus on the class LambdaMetafactoryand its capabilities. The next
section of this article assumes that you understand how metafactory methods work and what isMethodHandle


Tricks with lambda expressions


In this section, we show how to build dynamic constructions of lambda for use in daily tasks.


Checked exceptions and lambdas


It is no secret that all functional interfaces that are in Java do not support checked exceptions. The advantages of the checked exceptions over the usual ones are a very long-standing (and still hot) dispute.


And what if you need to use code with checked exceptions inside lambda expressions in combination with Java Streams? For example, you need to convert a list of strings into a list of URLs like this:


Arrays.asList("http://localhost/", "https://github.com").stream()
        .map(URL::new)
        .collect(Collectors.toList())

In the URL constructor (String), a checked exception is declared, so it cannot be used directly as a reference to a method in the Functiion class .


You say: "No, maybe if you use this trick":


publicstatic <T> T uncheckCall(Callable<T> callable){
    try { return callable.call(); }
    catch (Exception e) { return sneakyThrow(e); }
}
privatestatic <E extends Throwable, T> T sneakyThrow0(Throwable t)throws E { throw (E)t; }
publicstatic <T> T sneakyThrow(Throwable e){
    return Util.<RuntimeException, T>sneakyThrow0(e);
}
// Пример использования//return s.filter(a -> uncheckCall(a::isActive))//        .map(Account::getNumber)//        .collect(toSet());

This is a dirty hack. And that's why:


  • A try-catch block is used.
  • The exception is thrown again.
  • The dirty use of type erasure in java.

The problem can be solved in a more "legal" way, using knowledge of the following facts:


  • Checked exceptions are recognized only at the Java compiler level.
  • A section throwsis just metadata for a method without a semantic meaning at the JVM level.
  • Checked and normal exceptions are indistinguishable at the bytecode level in the JVM.

The solution is to wrap the method Callable.callin a method without a section throws:


static <V> V callUnchecked(Callable<V> callable){
    return callable.call();
}

This code will not compile because the method has Callable.callchecked exception exceptions in the section throws. But we can remove this section using a dynamically constructed lambda expression.


First we need to declare a functional interface in which there is no section throws
but which can delegate the call to Callable.call:


@FunctionalInterfaceinterfaceSilentInvoker{
    MethodType SIGNATURE = MethodType.methodType(Object.class, Callable.class);//сигнатура метода INVOKE
    <V> V invoke(final Callable<V> callable);
}

The second step is to create an implementation of this interface using LambdaMetafactoryand delegate the method call to the SilentInvoker.invokemethod Callable.call. As mentioned earlier, the section is throwsignored at the bytecode level, so the method SilentInvoker.invokecan call the method Callable.callwithout declaring exceptions:


privatestaticfinal SilentInvoker SILENT_INVOKER;
final MethodHandles.Lookup lookup = MethodHandles.lookup();
final CallSite site = LambdaMetafactory.metafactory(lookup,
                    "invoke",
                    MethodType.methodType(SilentInvoker.class),
                    SilentInvoker.SIGNATURE,
                    lookup.findVirtual(Callable.class, "call", MethodType.methodType(Object.class)),
                    SilentInvoker.SIGNATURE);
SILENT_INVOKER = (SilentInvoker) site.getTarget().invokeExact();

Third, we will write an auxiliary method that calls Callable.callwithout an exception declaration:


publicstatic <V> V callUnchecked(final Callable<V> callable)/*no throws*/{
    return SILENT_INVOKER.invoke(callable);
}

Now you can rewrite stream without any problems with checked exceptions:


Arrays.asList("http://localhost/", "https://dzone.com").stream()
        .map(url -> callUnchecked(() -> new URL(url)))
        .collect(Collectors.toList());

This code will compile without problems, because there are callUncheckedno declarations of checked exceptions. Moreover, calling this method can be inline using monomorphic inline caching , because it is only one class in the entire JVM that implements the interfaceSilentOnvoker


If the implementation Callable.callthrows an exception at runtime, then it will be intercepted by the calling function without any problems:


try{
    callUnchecked(() -> new URL("Invalid URL"));
} catch (final Exception e){
    System.out.println(e);
}

Despite the possibilities of this method, you should always remember the following recommendation:



Hide checked exceptions with callUnchecked only if you are sure that the code you are calling will not throw out any exceptions.



The following example shows an example of this approach:


callUnchecked(() -> new URL("https://dzone.com")); //этот URL всегда правильный и конструктор никогда не выкинет MalformedURLException

The full implementation of this method is here , it is part of the open source project SNAMP .


We work with Getters and Setters


This section will be useful to those who write serialization / deserialization for various data formats, such as JSON, Thrift, etc. Moreover, it can be quite useful if your code relies heavily on reflection for Getters and Setters in JavaBeans.


A getter declared in a JavaBean is a method with a name getXXXwithout parameters and a return data type other than void. A setter declared in a JavaBean is a method with a name setXXX, with one parameter, and a return void. These two notations can be represented as functional interfaces:


  • Getter can be represented by the Function class , in which the argument is a value this.
  • Setter can be represented by the BiConsumer class , in which the first argument is this, and the second is the value that is passed to Setter.

Now we will create two methods that can convert any getter or setter into these
functional interfaces. Never mind that both interfaces are generics. After erasing the types, the
actual data type will be Object. Automatic casting of the return type and arguments can be done with LambdaMetafactory. In addition, the Guava library will help with caching lambda expressions for identical getters and setters.


First step: you need to create a cache for getters and setters. The Method class from the Reflection API represents a real getter or setter and is used as the key.
The cache value is a dynamically constructed functional interface for a specific getter or setter.


privatestaticfinal Cache<Method, Function> GETTERS = CacheBuilder.newBuilder().weakValues().build();
privatestaticfinal Cache<Method, BiConsumer> SETTERS = CacheBuilder.newBuilder().weakValues().build();

Second, create factory methods that create an instance of a functional interface based on getter or setter references. 


privatestatic Function createGetter(final MethodHandles.Lookup lookup,
                                         final MethodHandle getter)throws Exception{
        final CallSite site = LambdaMetafactory.metafactory(lookup, "apply",
                MethodType.methodType(Function.class),
                MethodType.methodType(Object.class, Object.class), //signature of method Function.apply after type erasure
                getter,
                getter.type()); //actual signature of gettertry {
            return (Function) site.getTarget().invokeExact();
        } catch (final Exception e) {
            throw e;
        } catch (final Throwable e) {
            thrownew Error(e);
        }
}
privatestatic BiConsumer createSetter(final MethodHandles.Lookup lookup,
                                           final MethodHandle setter)throws Exception {
        final CallSite site = LambdaMetafactory.metafactory(lookup,
                "accept",
                MethodType.methodType(BiConsumer.class),
                MethodType.methodType(void.class, Object.class, Object.class), //signature of method BiConsumer.accept after type erasure
                setter,
                setter.type()); //actual signature of settertry {
            return (BiConsumer) site.getTarget().invokeExact();
        } catch (final Exception e) {
            throw e;
        } catch (final Throwable e) {
            thrownew Error(e);
        }
}

Automatic type casting between type arguments Objectin functional interfaces (after erasing types) and actual argument types and return values ​​is achieved using the difference between samMethodTypeand instantiatedMethodType(third and fifth arguments of the metafactory method, respectively). The type of the created instance of the method is the specialization of the method that provides the implementation of the lambda expression.


Third, create a facade for these factories with caching support: 


publicstatic Function reflectGetter(final MethodHandles.Lookup lookup, final Method getter)throws ReflectiveOperationException {
        try {
            return GETTERS.get(getter, () -> createGetter(lookup, lookup.unreflect(getter)));
        } catch (final ExecutionException e) {
            thrownew ReflectiveOperationException(e.getCause());
        }
}
publicstatic BiConsumer reflectSetter(final MethodHandles.Lookup lookup, final Method setter)throws ReflectiveOperationException {
        try {
            return SETTERS.get(setter, () -> createSetter(lookup, lookup.unreflect(setter)));
        } catch (final ExecutionException e) {
            thrownew ReflectiveOperationException(e.getCause());
        }
}

Method information obtained from a class instance Methodusing the Java Reflection API can be easily converted to MethodHandle. Bear in mind that the class instance methods always have a hidden first argument used to pass thisto this method. Static methods have no such parameter. For example, the actual method signature Integer.intValue()looks like int intValue(Integer this). This trick is used in our implementation of functional wrappers for getters and setters.


And now - time to test the code:


final Date d = new Date();
final BiConsumer<Date, Long> timeSetter = reflectSetter(MethodHandles.lookup(), Date.class.getDeclaredMethod("setTime", long.class));
timeSetter.accept(d, 42L); //the same as d.setTime(42L);final Function<Date, Long> timeGetter = reflectGetter(MethodHandles.lookup(), Date.class.getDeclaredMethod("getTime"));
System.out.println(timeGetter.apply(d)); //the same as d.getTime()//output is 42

This approach with cached getters and setters can be effectively used in libraries for serialization / deserialization (such as Jackson), which getters and setters use during serialization and deserialization.



Calling functional interfaces with dynamically generated implementations using LambdaMetaFactorymuch faster than calling through the Java Reflection API



The full version of the code can be found here , it is part of the SNAMP library .


Limitations and bugs


In this section, we will look at some of the bugs and limitations associated with lambda expressions in the Java compiler and the JVM. All these limitations can be reproduced in OpenJDK and Oracle JDK from javacversion 1.8.0_131 for Windows and Linux.


Creating lambda expressions from method handlers


As you know, a lambda expression can be constructed dynamically using LambdaMetaFactory. To do this, you need to define a handler — a class MethodHandlethat points to the implementation of a single method that is defined in the functional interface. Let's take a look at this simple example:


finalclassTestClass{
            String value = "";
            public String getValue(){
                return value;
            }
            publicvoidsetValue(final String value){
                this.value = value;
            }
        }
final TestClass obj = new TestClass();
obj.setValue("Hello, world!");
final MethodHandles.Lookup lookup = MethodHandles.lookup();
final CallSite site = LambdaMetafactory.metafactory(lookup,
                "get",
                MethodType.methodType(Supplier.class, TestClass.class),
                MethodType.methodType(Object.class),
                lookup.findVirtual(TestClass.class, "getValue", MethodType.methodType(String.class)),
                MethodType.methodType(String.class));
final Supplier<String> getter = (Supplier<String>) site.getTarget().invokeExact(obj);
System.out.println(getter.get());

This code is equivalent to:


final TestClass obj = new TestClass();
obj.setValue("Hello, world!");
final Supplier<String> elementGetter = () -> obj.getValue();
System.out.println(elementGetter.get());

But what if we replace the handler of the method that points to getValuethe handler that represents the getter of the field:


final CallSite site = LambdaMetafactory.metafactory(lookup,
                "get",
                MethodType.methodType(Supplier.class, TestClass.class),
                MethodType.methodType(Object.class),
                lookup.findGetter(TestClass.class, "value", String.class), //field getter instead of method handle to getValue
                MethodType.methodType(String.class));

This code should, expectedly, work because it findGetterreturns a handler that points to the getter of the field and has the correct signature. But, if you run this code, you will see the following exception:


java.lang.invoke.LambdaConversionException: Unsupported MethodHandle kind: getField

Interestingly, getter for the field works fine if we use MethodHandleProxies :


final Supplier<String> getter = MethodHandleProxies
                                       .asInterfaceInstance(Supplier.class, lookup.findGetter(TestClass.class, "value", String.class)
                                       .bindTo(obj));

It should be noted that it MethodHandleProxiesis not a good way to dynamically create lambda expressions, because this class simply wraps MethodHandlethe proxy class and delegates the call InvocationHandler.invoke to the MethodHandle.invokeWithArguments method . This approach uses Java Reflection and is very slow.


As shown earlier, not all method handlers can be used to create lambda expressions during code execution.



Only a few types of method handlers can be used to dynamically create lambda expressions.



Here they are:


  • REF_invokeInterface: can be created using Lookup.findVirtual for interface methods
  • REF_invokeVirtual: can be created using Lookup.findVirtual for virtual class methods
  • REF_invokeStatic: created using Lookup.findStatic for static methods
  • REF_newInvokeSpecial: can be created using the Lookup.findConstructor for constructors
  • REF_invokeSpecial: can be created using Lookup.findSpecial
    for private methods and early binding to class virtual methods

The remaining types of handlers will cause an error LambdaConversionException.


Generic exceptions


This bug is related to the Java compiler and the ability to declare generic exceptions in the section throws. The following code example demonstrates this behavior:


interfaceExtendedCallable<V, EextendsException> extendsCallable<V>{
        @OverrideV call()throws E;
}
final ExtendedCallable<URL, MalformedURLException> urlFactory = () -> new         
    URL("http://localhost");
    urlFactory.call();

This code must compile because the class constructor URLthrows it away MalformedURLException. But it does not compile. You receive the following error message:


Error:(46, 73) java: call() in <anonymous Test$CODEgt; cannot implement call() in ExtendedCallable
overridden method does not throw java.lang.Exception

But, if we replace the lambda expression with an anonymous class, then the code will compile:


final ExtendedCallable<URL, MalformedURLException> urlFactory = new ExtendedCallable<URL, MalformedURLException>() {
            @Overridepublic URL call()throws MalformedURLException {
                returnnew URL("http://localhost");
            }
        };
urlFactory.call();

Therefore:



Type inference for generic exceptions does not work correctly in combination with lambda expressions



Limitations of the type of parameterization


You can construct a generic object with several type constraints using the &: character <T extends A & B & C & ... Z>.
This method of determining generic parameters is rarely used, but in a certain way affects lambda expressions in Java due to some limitations:


  • Each type constraint other than the first must be an interface.
  • The pure version of the class with such generic only takes into account the first type constraint from the list.

The second constraint leads to different code behavior at compile time and at run time, when binding to the lambda expression occurs. This difference can be demonstrated using the following code:


finalclassMutableIntegerextendsNumberimplementsIntSupplier, IntConsumer{ //mutable container of int valueprivateint value;
    publicMutableInteger(finalint v){
        value = v;
    }
    @OverridepublicintintValue(){
        return value;
    }
    @OverridepubliclonglongValue(){
        return value;
    }
    @OverridepublicfloatfloatValue(){
        return value;
    }
    @OverridepublicdoubledoubleValue(){
        return value;
    }
    @OverridepublicintgetAsInt(){
        return intValue();
    }
    @Overridepublicvoidaccept(finalint value){
        this.value = value;
    }
}
static <T extends Number & IntSupplier> OptionalInt findMinValue(final Collection <T> values){
    return values.stream().mapToInt(IntSupplier::getAsInt).min();
}
final List <MutableInteger> values = Arrays.asList(new MutableInteger(10), new MutableInteger(20));
finalint mv = findMinValue(values).orElse(Integer.MIN_VALUE);
System.out.println(mv);

This code is absolutely correct and successfully compiled. The class MutableIntegersatisfies the constraints of the generic type T:


  • MutableIntegerinherited from Number.
  • MutableIntegerimplements IntSupplier.

But the code will fall with the exception at runtime:


java.lang.BootstrapMethodError: call site initialization exception
    at java.lang.invoke.CallSite.makeSite(CallSite.java:341)
    at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307)
    at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297)
    at Test.minValue(Test.java:77)
Caused by: java.lang.invoke.LambdaConversionException: Invalid receiver type classjava.lang.Number; not a subtype of implementation type interfacejava.util.function.IntSupplieratjava.lang.invoke.AbstractValidatingLambdaMetafactory.validateMetafactoryArgs(AbstractValidatingLambdaMetafactory.java:233)
    atjava.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:303)
    atjava.lang.invoke.CallSite.makeSite(CallSite.java:302)

This happens because the JavaStream pipeline captures only the pure type, which, in our case, is a class Numberand it does not implement the interface IntSupplier. This problem can be fixed by explicitly declaring the type of the parameter in a separate method used as a reference to the method:


privatestaticintgetInt(final IntSupplier i){
        return i.getAsInt();
}
privatestatic <T extends Number & IntSupplier> OptionalInt findMinValue(final Collection<T> values){
        return values.stream().mapToInt(UtilsTest::getInt).min();
}

This example demonstrates incorrect type inference in the compiler and runtime.



Handling several generic parameter type constraints in conjunction with the use of lambda expressions during compilation and execution is inconsistent


Tags:

Also popular now: