Make friends CI, unit tests and database

  • Tutorial
image

The article is about testing the interaction with the database in CI. I saw several solutions using docker and testcontainers, but I have my own and I want to share it.

My past java project was closely tied to a database. Long processing with retries, multithreading, and lock locks. For the task it was required to correct a couple of tricky SQL queries. I was somehow used to covering my code with tests, but before that all the SQL was reduced to primitive queries and it could be run on an H2 base in memory. And here on hardcore.

I could test simple SQL with my own hands and cowardly hammer on self-tests, justifying myself “I’m not some kind of bagodel, there are mistakes in simple code.” Actually, errors appear less often due to the availability of tests. It was possible to push responsibility to the testers - if I made a mistake somewhere, they would find it.

image

According to the ideology of unit testing, tests are needed only for testing individual modules, and if the module uses something from the outside, then this should be replaced with a stub. In practice, when it becomes too difficult to implement a stub, the module is simply ignored. Speed ​​and modularity are important, but it’s more important not to leave the code untested, even if it does not appear in coverage metrics. Therefore, the module is no longer considered to be a separate class, but a connected group, together with the configuration. The main thing with such statements is not to get to the concept of unit stretching to the cluster level.

For local testing, each developer has his own scheme, but tests are run on Jenkins and there is no connection to the database there yet. CI needs a separate circuit, it seems to be obvious. But on an empty diagram, it’s not very correct to run tests; creating a database structure in each test is time-consuming and fraught with a discrepancy between the structure in the test and in battle. Run on a pre-prepared base - I get a bunch of problems with branches. You can prepare the database before running all the tests using liquibase, first clearing everything to zero, and then updating to the latest version.

Rollback is often forgotten to finalize and you have to clean the bases with your hands on test environments. Test it and him! The algorithm is as follows:

  1. remove everything under the root (for the purity of the experiment)
  2. update to latest version
  3. perform rollback 1-2 versions back
  4. update to the latest version (tests need to be driven on the new database structure, plus checking that rollback did not forget to delete anything, which will prevent the update from rolling again)

Fellow developers do not want to run rollback testing when starting each test. We make the switch.

    project.ext.doRollbackTest = { Boolean.parseBoolean(localConfig['test.rollback.enabled'] as String) }

While there is training on cats, everything is ok. But a dynamically developing project makes its own adjustments - 2 pull-quests, simultaneous assembly. One instance is testing something, and the second is knocking the base from under its feet. It is solved by a simple ban on parallel assemblies.

image

And again a fall - I decided to run tests using the Jenkins account, because everything is ok on the personal one, and the pool of requests falls for unclear reasons. We recall the fervent talk that DevOps is a culture and the use of technical accounts for personal purposes is unacceptable.

image

Accumulated 10 pool of requests. All collected, redistributed, you can merge. The first one went, the main branch changed - the rest together stand in line for rebuilding. You can merge as you progress, but there are priorities. More urgent pullrequests, less urgent, are also hanging up due to errors in the code. In general, parallelize back.

The assembly should be simple, consist of simple steps and be understandable even to yesterday’s student. In 99% there is no problem in that the assembly of the request and release pool is sequential, and not parallel. If the review does not accumulate more than 1-2 PR, then the prohibition of simultaneous assemblies is quite enough.

And for parallel launch we need bases or schemes to which each launched test will have exclusive access.

Option one is to allocate dynamically. Creating a schema in the database is fast. Having a cloud with an API, you can allocate a database there.

If you do not work out the deletion of old databases, you can quickly finish the disk space when the tests fall and “forget” to free up resources. It is “when,” not “if.”

Option two - a database / schema pool with a separate management service. Outside the API sticks out, give the Base for the Time, take back the Free Base Before the Term. What it will return: a dedicated server with a database or just a little schema, it does not matter. The main thing is that the resource will not be lost forever.

Option three - a pool of bases / schemes for self-regulation. A resource is needed for the exchange of information about locks and the database pool itself.

I settled on the latter option, since it is easier for me to fasten it and it is not particularly required to support it. The logic is as follows - several (10 for example) circuits are created and all the necessary information about connecting to them is added to the shared resource, each test instance makes a start mark before starting it, and after the end it deletes it. If the test crashes before finalizing, the circuit will be considered free at the end of the timeout.

Reading settings:

    project.ext.localConfig = new Properties()
    localConfig.load(file("${rootDir}/local.properties").newReader())

Working with sql from gradle scripts requires driver loading:
    configurations {
        driver
    }
    dependencies {
        driver group: "oracle", name: "ojdbc6", version: "11.+"
    }
    task initDriver {
        doLast {
            ClassLoader loader = GroovyObject.class.classLoader
            configurations.driver.each { File file ->
                loader.addURL(file.toURL())
            }
        }
    }

Connection:

    import groovy.sql.Sql
    project.ext.createSqlInstance = {
        return Sql.newInstance(
                url: localConfig["pool.db.url"],
                user: localConfig["pool.db.username"],
                password: localConfig["pool.db.password"],
                driver: localConfig["pool.db.driverClass"])
    }

Information exchange can be carried out through the database table. Initialization of the reference table (it should work once, then the table lives until the end of time):

    task initDbPool {
        dependsOn initDriver
        doLast {
            Integer poolSize = 10
            Sql sql = createSqlInstance() as Sql
            String tableName = localConfig["pool.db.referenceTable"]
            String baseName = localConfig["pool.db.baseName"]
            String basePass = localConfig["pool.db.basePass"]
            String token = "{id}"
            List tableExists = sql.rows("select table_name from all_tables where table_name=?", [tableName])
            assert tableExists.isEmpty()
            sql.execute("""
                CREATE TABLE ${tableName} (
                ID NUMBER(2) NOT NULL PRIMARY KEY,
                METADATA VARCHAR2(200) NOT NULL,
                PROCESSED TIMESTAMP NULL,
                GUID VARCHAR2(36) NULL)
            """, [])
            for (Integer i = 0 ; i < poolSize ; i++) {
                String username = baseName.replace(token, i.toString())
                String password = basePass.replace(token, i.toString())
                sql.execute("""
                    CREATE USER ${username}
                    IDENTIFIED BY "${password}"
                    DEFAULT TABLESPACE USERS
                    TEMPORARY TABLESPACE TEMP
                    PROFILE DEFAULT
                    QUOTA UNLIMITED ON USERS
                """, [])
                sql.execute("grant connect to ${username}", [])
                sql.execute("grant create sequence to ${username}", [])
                sql.execute("grant create session to ${username}", [])
                sql.execute("grant create table to ${username}", [])
                String metadata = JsonOutput.toJson([
                    "app.db.driverClass": localConfig["pool.db.driverClass"],
                    "app.db.url": localConfig["pool.db.url"],
                    "app.db.username": username,
                    "app.db.password": password
                        ])
                sql.execute("""
                    INSERT INTO ${tableName} (id, metadata)
                    values (?, ?)
                """, [i, metadata])
            }
        }
    }

Developers have their own schemes for debugging and assembly, so the use of the pool must be turned off:

    project.ext.isCiBuild = { Boolean.parseBoolean(localConfig['pool.db.enabled'] as String) }

Take and free base:

    task lockDb {
        dependsOn initDriver
        onlyIf isCiBuild
        doLast {
            project.ext.lockUid = UUID.randomUUID().toString()
            String tableName = localConfig["pool.db.referenceTable"]
            Sql sql = createSqlInstance() as Sql
            sql.executeUpdate("""UPDATE ${tableName} SET GUID = ?, PROCESSED = SYSDATE
                    WHERE ID IN (
                        SELECT ID FROM (
                            SELECT ID, ROW_NUMBER() OVER (ORDER BY PROCESSED) AS RN
                            FROM ${tableName} WHERE GUID IS NULL OR PROCESSED < (SYSDATE - NUMTODSINTERVAL(?, 'MINUTE'))
                        ) WHERE RN = 1
                    )
                    """, [lockUid, 15])
            def meta = sql.firstRow("SELECT METADATA FROM ${tableName} WHERE GUID = ?", [lockUid])
            assert meta != null, "No free databases in pool"
            def slurper = new JsonSlurper()
            Map metadata = slurper.parseText(meta["METADATA"] as String) as Map
            localConfig.putAll(metadata)
            logger.info("Database locked, {}", metadata)
        }
    }
    task unlockDb {
        dependsOn lockDb // init lockUid
        onlyIf isCiBuild
        doLast {
            try {
                String tableName = localConfig["pool.db.referenceTable"]
                Sql sql = createSqlInstance() as Sql
                sql.executeUpdate("UPDATE ${tableName} SET GUID = NULL WHERE GUID = ?",
                        [lockUid])
                logger.info("Database unlocked")
            } catch (ignored) {
                logger.error(ignored)
            }
        }
    }

If you complete the assembly 2 times in a row, different schemes can be selected and different values ​​will remain when assembling the property files. For local launches, the settings are static.

    configure([processResources, processTestResources]) { Task t ->
        if (project.ext.isCiBuild()) {
            t.outputs.upToDateWhen { false }
        }
        t.filesMatching('**/*.properties') {
            filter(ReplaceTokens, tokens: localConfig, beginToken: '${', endToken: '}')
        }
    }

Tasks for testing rollback:

    task restoreAfterRollbackTest(type: LiquibaseTask) {
        command = 'update'
    }
    task rollbackTest(type: LiquibaseTask) {
        dependsOn lockDb
        command = 'rollback'
        requiresValue = true
        doFirst {
            project.ext.liquibaseCommandValue = localConfig['test.rollback.tag']
        }
        doLast {
            project.ext.liquibaseCommandValue = null
        }
    }

And set up the execution order:

    configure([project]) {
        tasks.withType(LiquibaseTask.class) { LiquibaseTask t ->
            logger.info("Liquibase task {} must run after {}", t.getName(), configLiquibase.getName())
            (t as Task).dependsOn configLiquibase
            if (isCiBuild()) {
                logger.info("Liquibase task {} must run after {}", t.getName(), lockDb.getName())
                (t as Task).dependsOn lockDb
                (t as Task).finalizedBy unlockDb
            }
        }
        // На этапе CI:
        // 1. Чистим БД в 0 (dropAll)
        // 2. Обновляем БД для проверки rollback (update)
        // 3. Проверяем, что работает откат до изначального состояния (rollback tag)
        // 4. Обновляем БД для прогона тестов (update)
        // 5. Прогоняем тесты на БД
        if (doRollbackTest()) {
            def setTaskOrdering = { List lst ->
                for (int i = 0; i < lst.size() - 1; i++) {
                    logger.info("Task {} must run after {}", lst[i + 1].getName(), lst[i].getName())
                    lst[i + 1].dependsOn lst[i]
                }
            }
            setTaskOrdering([
                    lockDb,
                    configLiquibase,
                    dropAll,
                    update,
                    rollbackTest,
                    restoreAfterRollbackTest,
                    processTestResources,
                    test,
            ])
            lockDb.finalizedBy unlockDb
            test.finalizedBy unlockDb
        }
    }

Database allocation and rollback testing can be placed inside tests. Ways to run code before and after all tests have been completed: in Junit5, this is BeforeAllCallback, in TestNG BeforeSuite.

To the question “why should sql be tested by the java programmer” the answer is that any code should be tested. There are exceptions and some code is irrational to test at a given time.

I would like to know how the problem of testing the interaction with the database is solved by other programmers? Have containers arrived in every home, or is integration testing passed onto the shoulders of testers?

Full listing
import groovy.json.JsonOutput
import groovy.json.JsonSlurper
import groovy.sql.Sql
import org.apache.tools.ant.filters.ReplaceTokens
import org.liquibase.gradle.LiquibaseTask
plugins {
    id 'java'
    id 'org.liquibase.gradle' version '2.0.1'
}
configurations {
    driver
}
repositories {
    jcenter()
    mavenCentral()
    maven {
        url = "http://www.datanucleus.org/downloads/maven2/"
    }
}
dependencies {
    implementation 'com.google.guava:guava:27.0.1-jre'
    implementation 'org.springframework:spring-core:5.1.7.RELEASE'
    implementation 'org.springframework:spring-context:5.1.7.RELEASE'
    implementation 'org.springframework:spring-jdbc:5.1.7.RELEASE'
    testImplementation 'junit:junit:4.12'
    testImplementation 'org.springframework:spring-test:5.1.7.RELEASE'
    testRuntime 'oracle:ojdbc6:11.+'
    liquibaseRuntime 'org.liquibase:liquibase-core:3.6.1'
    liquibaseRuntime 'oracle:ojdbc6:11.+'
    liquibaseRuntime 'org.yaml:snakeyaml:1.24'
    driver group: "oracle", name: "ojdbc6", version: "11.+"
}
project.ext.localConfig = new Properties()
localConfig.load(file("${rootDir}/local.properties").newReader())
project.ext.isCiBuild = { Boolean.parseBoolean(localConfig['pool.db.enabled'] as String) }
project.ext.doRollbackTest = { Boolean.parseBoolean(localConfig['test.rollback.enabled'] as String) }
task configLiquibase {
    doLast {
        liquibase {
            activities {
                testdb {
                    changeLogFile 'changelog.yaml'
                    url localConfig['app.db.url']
                    driver localConfig['app.db.driverClass']
                    username localConfig['app.db.username']
                    password localConfig['app.db.password']
                    logLevel 'debug'
                    classpath "${project.projectDir}/db"
                    contexts 'main'
                }
                runList = 'testdb'
            }
        }
    }
}
task initDriver {
    doLast {
        ClassLoader loader = GroovyObject.class.classLoader
        configurations.driver.each { File file ->
            loader.addURL(file.toURL())
        }
    }
}
project.ext.createSqlInstance = {
    return Sql.newInstance(
            url: localConfig["pool.db.url"],
            user: localConfig["pool.db.username"],
            password: localConfig["pool.db.password"],
            driver: localConfig["pool.db.driverClass"])
}
task initDbPool {
    dependsOn initDriver
    doLast {
        Integer poolSize = 10
        Sql sql = createSqlInstance() as Sql
        String tableName = localConfig["pool.db.referenceTable"]
        String baseName = localConfig["pool.db.baseName"]
        String basePass = localConfig["pool.db.basePass"]
        String token = "{id}"
        List tableExists = sql.rows("select table_name from all_tables where table_name=?", [tableName])
        assert tableExists.isEmpty()
        sql.execute("""
            CREATE TABLE ${tableName} (
            ID NUMBER(2) NOT NULL PRIMARY KEY,
            METADATA VARCHAR2(200) NOT NULL,
            PROCESSED TIMESTAMP NULL,
            GUID VARCHAR2(36) NULL)
        """, [])
        for (Integer i = 0 ; i < poolSize ; i++) {
            String username = baseName.replace(token, i.toString())
            String password = basePass.replace(token, i.toString())
            sql.execute("""
                CREATE USER ${username}
                IDENTIFIED BY "${password}"
                DEFAULT TABLESPACE USERS
                TEMPORARY TABLESPACE TEMP
                PROFILE DEFAULT
                QUOTA UNLIMITED ON USERS
            """, [])
            sql.execute("grant connect to ${username}", [])
            sql.execute("grant create sequence to ${username}", [])
            sql.execute("grant create session to ${username}", [])
            sql.execute("grant create table to ${username}", [])
            String metadata = JsonOutput.toJson([
                "app.db.driverClass": localConfig["pool.db.driverClass"],
                "app.db.url": localConfig["pool.db.url"],
                "app.db.username": username,
                "app.db.password": password
                    ])
            sql.execute("""
                INSERT INTO ${tableName} (id, metadata)
                values (?, ?)
            """, [i, metadata])
        }
    }
}
task lockDb {
    dependsOn initDriver
    onlyIf isCiBuild
    doLast {
        project.ext.lockUid = UUID.randomUUID().toString()
        String tableName = localConfig["pool.db.referenceTable"]
        Sql sql = createSqlInstance() as Sql
        sql.executeUpdate("""UPDATE ${tableName} SET GUID = ?, PROCESSED = SYSDATE
                WHERE ID IN (
                    SELECT ID FROM (
                        SELECT ID, ROW_NUMBER() OVER (ORDER BY PROCESSED) AS RN
                        FROM ${tableName} WHERE GUID IS NULL OR PROCESSED < (SYSDATE - NUMTODSINTERVAL(?, 'MINUTE'))
                    ) WHERE RN = 1
                )
                """, [lockUid, 15])
        def meta = sql.firstRow("SELECT METADATA FROM ${tableName} WHERE GUID = ?", [lockUid])
        assert meta != null, "No free databases in pool"
        def slurper = new JsonSlurper()
        Map metadata = slurper.parseText(meta["METADATA"] as String) as Map
        localConfig.putAll(metadata)
        logger.info("Database locked, {}", metadata)
    }
}
task unlockDb {
    dependsOn lockDb // init lockUid
    onlyIf isCiBuild
    doLast {
        try {
            String tableName = localConfig["pool.db.referenceTable"]
            Sql sql = createSqlInstance() as Sql
            sql.executeUpdate("UPDATE ${tableName} SET GUID = NULL WHERE GUID = ?",
                    [lockUid])
            logger.info("Database unlocked")
        } catch (ignored) {
            logger.error(ignored)
        }
    }
}
configure([processResources, processTestResources]) { Task t ->
    if (project.ext.isCiBuild()) {
        t.outputs.upToDateWhen { false }
    }
    t.filesMatching('**/*.properties') {
        filter(ReplaceTokens, tokens: localConfig, beginToken: '${', endToken: '}')
    }
}
task restoreAfterRollbackTest(type: LiquibaseTask) {
    command = 'update'
}
task rollbackTest(type: LiquibaseTask) {
    dependsOn lockDb
    command = 'rollback'
    requiresValue = true
    doFirst {
        project.ext.liquibaseCommandValue = localConfig['test.rollback.tag']
    }
    doLast {
        project.ext.liquibaseCommandValue = null
    }
}
configure([project]) {
    tasks.withType(LiquibaseTask.class) { LiquibaseTask t ->
        logger.info("Liquibase task {} must run after {}", t.getName(), configLiquibase.getName())
        (t as Task).dependsOn configLiquibase
        if (isCiBuild()) {
            logger.info("Liquibase task {} must run after {}", t.getName(), lockDb.getName())
            (t as Task).dependsOn lockDb
            (t as Task).finalizedBy unlockDb
        }
    }
    // На этапе CI:
    // 1. Чистим БД в 0 (dropAll)
    // 2. Обновляем БД для проверки rollback (update)
    // 3. Проверяем, что работает откат до изначального состояния (rollback tag)
    // 4. Обновляем БД для прогона тестов (update)
    // 5. Прогоняем тесты на БД
    if (doRollbackTest()) {
        def setTaskOrdering = { List lst ->
            for (int i = 0; i < lst.size() - 1; i++) {
                logger.info("Task {} must run after {}", lst[i + 1].getName(), lst[i].getName())
                lst[i + 1].dependsOn lst[i]
            }
        }
        setTaskOrdering([
                lockDb,
                configLiquibase,
                dropAll,
                update,
                rollbackTest,
                restoreAfterRollbackTest,
                processTestResources,
                test,
        ])
        lockDb.finalizedBy unlockDb
        test.finalizedBy unlockDb
    }
}

pool.db.enabled=false
test.rollback.enabled=true
pool.db.driverClass=oracle.jdbc.driver.OracleDriver
pool.db.url=jdbc:oracle:thin:@localhost:1527:ORCLCDB
pool.db.username=SYSTEM
pool.db.password=Oradoc_db1
pool.db.referenceTable=c##test_user1.REF_TABLE
pool.db.baseName=C##CI_SCHEMA_{id}
pool.db.basePass=CI_SCHEMA_{id}_PASS
app.db.driverClass=
app.db.url=
app.db.username=
app.db.password=
test.rollback.tag=version_1


Also popular now: