Test plugins for Gradle correctly

    Somehow, when preparing one of the reports on the development of plugins for Gradle, the task arose - how to test our skills. Without tests, it’s generally bad to live, and when your code really runs in a separate process and even more so because you want debug, you want to run quickly and don’t want to write a million examples to test all possible cases. Under the cut, a comparison of several testing methods that we managed to try.


    Guinea pig


    Our experimental rabbit will be the project that we prepared with tolkkv for the JPoint 2016 conference . In short, we wrote a plug-in that will collect documentation from various projects and generate a regular html-document with cross-reference links. But it's not about how we wrote the plug-in itself (although it was also fun and exciting), but how to test what you write. I confess, but almost the entire project we tested integration, through examples. And at some point they realized that it was worth thinking about another way of testing. So, our candidates:



    The task is the same everywhere. Just check that our documentation plugin is connected and there is a task that can execute successfully. Chased.


    Gradle test kit


    Now it is in the incubation stage, which was very noticeable when we tried to screw it. If we take an example from the documentation and naively apply it to our realities (see the example below), then nothing will work. Let's figure out what we did.


    @Slf4j
    class TestSpecification extends Specification {
      @Rule
      final TemporaryFolder testProjectDir = new TemporaryFolder()
      def buildFile
      def setup() {
        buildFile = testProjectDir.newFile('build.gradle')
      }
      def "execution of documentation distribution task is up to date"() {
        given:
        buildFile << """
                  buildscript {
                    repositories { jcenter() }
                    dependencies {
                      classpath 'org.asciidoctor:asciidoctor-gradle-plugin:1.5.3'
                    }
                  }
                  apply plugin: 'org.asciidoctor.convert'
                  apply plugin: 'ru.jpoint.documentation'
                  docs {
                    debug = true
                  }
                  dependencies {
                    asciidoctor 'org.asciidoctor:asciidoctorj:1.5.4'
                    docs 'org.slf4j:slf4j-api:1.7.2'
                  }
              """
        when:
        def result = GradleRunner.create()
            .withProjectDir(testProjectDir.root)
            .withArguments('documentationDistZip')
            .build()
        then:
        result.task(":documentationDistZip").outcome == TaskOutcome.UP_TO_DATE
      }
    }

    We use Spock, although JUnit can also be used. Our project will lie and run in a temporary folder, which is defined through testProjectDir. In the setup method, we create a new project build file. In given, we defined the content of this file, connected to it the plugins and dependencies we needed. In the when section through the new class GradleRunner, we pass the previously defined directory with the project and say that we want to run the task from the plugin. In the then section, we check that we have the task, but since we have not defined any documents, we do not need to execute it.


    Duck, by running the test, we find out that the test framework does not know what kind of plug-in - ru.jpoint.documentation- we connected. Why it happens? Because now GradleRunnerthe classpath of the plugin does not pass inside itself. And this greatly limits us in testing. We go to the documentation and find out that there is a method withPluginClasspathin which we can transfer the resources we need, and they will be picked up during the testing process. It remains to understand how to form it.


    If you think this is obvious, think again. To solve the problem, you yourself need to create a text file with a set of resources in the builddirectory through a separate task (thanks to Gradle for the imperative approach) . We write:


    task createClasspathManifest {
        def outputDir = sourceSets.test.output.resourcesDir
        inputs.files sourceSets.main.runtimeClasspath
        outputs.dir outputDir
        doLast {
            outputDir.mkdirs()
            file("$outputDir/plugin-classpath.txt").text = sourceSets.main.runtimeClasspath.join("\n")
        }
    }

    We start, we receive a file. Now we go to our test and in setup we add the following code, pleasant for reading:


        def pluginClasspathResource = getClass().classLoader.findResource("plugin-classpath.txt")
        if (pluginClasspathResource == null) {
          throw new IllegalStateException("Did not find plugin classpath resource, run `testClasses` build task.")
        }
        pluginClasspath = pluginClasspathResource.readLines()
            .collect { new File(it) }

    Now pass classpathin GradleRunner. Run, and nothing works. We go to the forums and find out that this only works with Gradle 2.8+. We check that we have 2.12 and are sad. What to do? Let's try to do as advised to do for Gradle 2.7 and below. We ourselves will create another one classpathand add it directly to buildscript:


    def classpathString = pluginClasspath
            .collect { it.absolutePath.replace('\\', '\\\\') }
            .collect { "'$it'" }
            .join(", ")

    dependencies {
        classpath files($classpathString)
        ...
    }

    Launch - it works. These are not all the problems. You can read the epic trade and it will become completely sad.


    2.13 update: when we experimented, a new version has not yet been released. It fixed (finally) a problem with pulling up resources and now the code looks much more decent and noble. To do this, you need to connect the plugin a little differently:


    plugins {
        id 'ru.jpoint.documentation'
    }

    and run GradleRunnerwith an empty classpath:


    def result = GradleRunner.create()
            .withProjectDir(testProjectDir.root)
            .withArguments('documentationDistZip')
            .withPluginClasspath()
            .build()

    All that remains is the chagrin that Idea cannot run this test through the context menu, because it does not know how to correctly substitute the necessary resources. Through ./gradleweverything works great.


    Fix from d10xa : In the settings, select "Gradle Test Runner" (Settings-> Build-> Build Tools-> Gradle-> Runner), delete the existing configuration and run the test again.


    Bottom line: the direction is right, but sometimes it hurts.


    Nebula test


    The second candidate showed himself much better. All you need to do is connect the plugin to your dependencies:


    functionalTestCompile 'com.netflix.nebula:nebula-test:4.0.0'

    Then, in the specification, we can create a build.gradlefile by analogy with the previous example :


    def setup() {
        buildFile << """
                buildscript {
                  repositories { jcenter() }
                  dependencies {
                    classpath 'org.asciidoctor:asciidoctor-gradle-plugin:1.5.3'
                  }
                }
                apply plugin: 'org.asciidoctor.convert'
                apply plugin: info.developerblog.documentation.plugin.DocumentationPlugin
                docs {
                  debug = true
                }
                dependencies {
                  asciidoctor 'org.asciidoctor:asciidoctorj:1.5.4'
                  docs 'org.slf4j:slf4j-api:1.7.2'
                }
    """
      }

    But the test itself looks easy, understandable, and most importantly - it runs without squats:


    def "execution of documentation distribution task is success"() {
        when:
        createFile("/src/docs/asciidoc/documentation.adoc")
        ExecutionResult executionResult = runTasksSuccessfully('documentationDistZip')
        then:
        executionResult.wasExecuted('documentationDistZip')
        executionResult.getSuccess()
    }

    In this example, we also created a file with documentation, and therefore the result of the execution of our task will be SUCCESS.


    Bottom line: everything is very cool. Recommended for use.


    Unit testing


    Ok, all we did before was all these integration tests. Let's see what we can do through the unit-test mechanism.


    First, we configure the project simply through the code:


    def setup() {
            project = new ProjectBuilder().build()
            project.buildscript.repositories {
                jcenter()
            }
            project.buildscript.dependencies {
                classpath 'org.asciidoctor:asciidoctor-gradle-plugin:1.5.3'
            }
            project.plugins.apply('org.asciidoctor.convert')
            project.plugins.apply(DocumentationPlugin.class)
            project.dependencies {
                asciidoctor 'org.asciidoctor:asciidoctorj:1.5.4'
                docs 'org.slf4j:slf4j-api:1.7.2'
            }
        }

    As you can see, this is practically no different from what we wrote before, only they Closureare written a little longer.


    Now we can test that our task from the plug-in really appeared in the configured project (and in general the configuration was successful):


    def "execution of documentation distribution task is success"() {
            when:
            project
            then:
            project.getTasksByName('documentationDistZip', true).size() == 1
    }

    But we cannot test more of this. That is, through this method we cannot understand that task will do what it is supposed to, and, say, the document will really be formed.


    Bottom line: can be used to verify the configuration of projects. This is faster than testing through actual execution. But our possibilities are very limited.


    Summary


    Recommended use Nebula Testfor testing plugins. If you have branchy logic when configuring a project, then it makes sense to look towards Unit testing. Well, we are waiting for the finished one Gradle Test Kit.


    Link to the project with tests and a plugin: https://github.com/aatarasoff/documentation-plugin-demo


    Also popular now: