Code generation for the backend. What to generate, how and why?

    In this article I want to show how a backend (and a bit of front-end) code is generated in our company, why is it needed at all and how to do it better.

    What exactly we will generate is not so important.
    It is important that we describe 3 types of objects on the basis of which we will generate front-end interaction with the backend, and in some places the backend implementation will be fully implemented.

    These types of objects:
    1. Messages - objects that, when serialized in JSON, participate in the exchange of information between the front-end and the backend
    . 2. Endpoints - A URI that calls the frontend along with a description of the HTTP method, request parameters, Request Body type, and responder type
    3. Entities- These are messages for which they have standard endpoints for Create / Update / List / Delete (maybe not all), they are stored in the database and for them there is a Data Access Object, or Spring JPA repository - in general it depends on the technology, but

    I don’t do any kind of access to the database at all, but
    1) I know that it is written in Typescript, so we also generate timecode classes
    2) Most of the requirements for the backend come from frontend developers.

    Code requirements



    So, what are the requirements from the front end?

    1. REST-like interaction interface
    2. Monotonous responses - json, payload in the 'data' field
    3. Monotonous errors if an exception occurred on the backend, it is also advisable to add stack trace
    4. “correct” HTTP codes - 404 if the book is not found, 400 if bad request (say not valid json) etc.

    I’ll add requirements to the backend code “on my own”:
    1. Error handling in one place
    2. Ability to stop flow anywhere in the code and return the necessary HTTP code
    3. I want to write some business logic as blocking, and some as asynchronous, depending on used libraries. But all this should work in one asynchronous framework
    4. It is advisable that the backend developers do not think about HTTP requests and responses, about Vertx routes and event basses, but simply write their business logic.

    It is advisable to implement all of the above requirements with inheritance and composition, and only where it does not work, use code generation.

    It is also desirable to generate parallel classes for time script and kotlin so that the front end always sends the backend what is needed (and not rely on developers that they will not forget to add to the class a new field)

    What will we generate



    For example, take a hypothetical web application that can save and edit books, display their list and search by name.

    In terms of technology, the Kotlin backend, Vert.x, coroutines. Something like what I showed in the article “Three Paradigms of Asynchronous Programming in Vertx”

    To make it more interesting, we will make access to the database based on Spring Data JPA.
    I'm not saying that you need to mix Spring and Vert.x in one project (although I do it myself, I admit), but I just take Spring since it is easiest for him to show generation based on Entities.

    Project structure with generation



    Now you need to make a project for generation.
    We will have many gradle projects. Now I’ll make them in one git repository, but in real life everyone should sit in his own, because they will change at different times, they will have their own versions.

    So, the first project is a project with annotations that will indicate our routers, HTTP methods, etc. Let's call it metainfo
    Two other projects depend on it:
    codegen and api

    api contain descriptions of routers and messages - those classes that will go back and forth between the backend and front-end
    codegen - a code generation project (but not a project in which code is generated!) - it contains collecting information from apiclasses and actually code generators.
    The generators will receive all the details of the generation in arguments - from which package to take the description of the routers, to which directory to generate, what name of the Velocity template for generation - i.e. metainfo and codegen can generally be used in completely different projects.

    Well, there are two projects in which generation will actually be carried out:
    frontend-generated in which we will generate the Typescript class that correspond to our kotlin messages
    and backend - with the Vertx application itself.

    In order for one project to “see” the result of compiling another, we will use the plugin to publish artifacts to the local Maven repository.

    Projectmetafinfo :

    Annotations with which we will mark sources of generation - descriptions of endpoins, messages, entities:
    /* Contains a number of endpoints. We will generate Vert.x router or Spring MVC controller from it*/
    annotation class EndpointController(val url:String)
    /* Endpoint inside a controller. Concrete URI and HTTP method. May be has query param */
    annotation class Endpoint(val method: HttpMethodName, val param: String = "")
    /* For empty constructor generation */
    annotation class EmptyConstructorMessage
    /* Make abstract implementation method for endpoint logic asynchronous */
    annotation class AsyncHandler
    /* All the next annotations are for Entities only:*/
    annotation class GenerateCreate
    annotation class GenerateUpdate
    annotation class GenerateGetById
    annotation class GenerateList
    annotation class GenerateDelete
    /* Make CRUD implementation abstract, so that we will override it*/
    annotation class AbstractImplementation
    /* Generate search by this field in DAO layer */
    annotation class FindBy
    /* This entity is child of another entity, so generate end point like
     /parent/$id/child to bring all children of concrete parent
     instead of
     /child - bring all entities of this type
     */
    annotation class ChildOf(vararg val parents: KClass<*>)
    enum class HttpMethodName {
        POST,PUT,GET,DELETE
    }
    

    For Typescript classes, we will define annotations that can be hung on the fields and which will fall into the generated Typescript class
    annotation class IsString
    annotation class IsEmail
    annotation class IsBoolean
    annotation class MaxLength(val len:Int)
    


    The source code of the project metainfo

    project api :

    Notice the Addons noArg and jpa in build.gradle to generate a no-argument constructor

    Fantasy I do not have, so we create what is insane controller descriptions and Entities for our application:

    @EndpointController("/util")
    interface SearchRouter {
        @Endpoint(HttpMethodName.GET, param = "id")
        fun search(id: String): String
        @Endpoint(method = HttpMethodName.POST)
        @AsyncHandler
        fun search(searchRequest: SearchRequest) // we have no check or response type
    }
    data class SearchRequest(
        @field:IsString
        val author: String?,
        @field:IsEmail
        val someEmail: String,
        @field:IsString
        val title: String?
    )
    @GenerateList
    @GenerateGetById
    @GenerateUpdate
    @Entity
    @AbstractImplementation
    data class Book(
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        var id: Long?,
        @field:IsBoolean
        @Column(name = "is_deleted")
        var hardcover: Boolean,
        @field:IsString
        @field:MaxLength(128)
        @Column(nullable = false, length = 128)
        val title: String,
        @field:IsString
        @field:MaxLength(128)
        @Column(nullable = false, length = 255)
        val author: String
    )
    @GenerateList
    @GenerateGetById
    @GenerateUpdate
    @GenerateDelete
    @GenerateCreate
    @Entity
    @ChildOf(Book::class)
    data class Chapter(
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        var id: Long?,
        @Column(nullable = false, name = "book_id")
        var bookId: Long?,
        @field:IsString
        @field:MaxLength(128)
        @Column(nullable = false, length = 128)
        @field:FindBy
        val name: String,
        @Column(nullable = false)
        val page:Int
    )


    The source code of the api

    project The codegen project :

    First we define the “descriptors” - those classes that we will fill out by going through the reflection on our api project:

    data class EndPoint(
        val url: String, val input: String?, val param: String?, val output: String, val method: String,
        val handler: String, val asyncHandler: Boolean
    )
    data class Router(val name: String, val url: String, val endpoints: List)
    class Entity(
        name: String, val parents: List, val abstractVerticle: Boolean,
        val crudFeatures: CrudFeatures, fields: List, var children: List
    ) : Message(name, fields) {
        fun shouldGenerateRouterAndVerticle(): Boolean {
            return crudFeatures.generateRouterAndVerticle()
        }
        override fun toString(): String {
            return "Entity(parents=$parents, abstractVerticle=$abstractVerticle, crudFeatures=$crudFeatures, children=$children)"
        }
    }
    data class CrudFeatures(
        val list: Boolean, val create: Boolean, val update: Boolean, val delete: Boolean,
        val get: Boolean
    ) {
        fun generateRouterAndVerticle(): Boolean {
            return list || create || update || delete || get
        }
    }
    open class Message(val name: String, val fields: List)
    data class Field(val name: String, val type: String, val validators: List, val findBy: Boolean)
    


    The code that collects the information looks like this:
    class EntitiesCreator(typeMapper: TypeMapper, frontendAnnoPackage:String) {
        private val messagesDescriptor =
            MessagesCreator(typeMapper, frontendAnnoPackage)
        fun createEntities(entitiesPackage: String): List {
            val reflections = Reflections(entitiesPackage, SubTypesScanner(false))
            val types = reflections.getSubTypesOf(Object::class.java)
            return types.map { createEntity(it) }
        }
        fun createEntityRestEndpoints(entity: Entity): List {
            val name = entity.name
            val url = name.toLowerCase()
            val endpoints: MutableList = mutableListOf()
            if (entity.crudFeatures.create) {
                endpoints.add(
                    EndPoint(url, name, null, name, "post", "handleNew$name", false)
                )
            }
            if (entity.crudFeatures.get) {
                endpoints.add(
                    EndPoint(
                        "$url/:id", null, "id", name, "get", "handleGet$name", false
                    )
                )
            }
            if (entity.crudFeatures.update) {
                endpoints.add(
                    EndPoint(url, name, null, name, "put", "handleUpdate$name", false)
                )
            }
            if (entity.crudFeatures.delete) {
                endpoints.add(
                    EndPoint(
                        "$url/:id", null, "id", "", "delete", "handleDelete$name", false
                    )
                )
            }
            if (entity.crudFeatures.list) {
                if (entity.parents.isEmpty()) {
                    endpoints.add(
                        EndPoint(
                            url, null, null, "List<$name>", "get", "handleGetAllFor$name", false
                        )
                    )
                }
            }
            entity.children.forEach {
                endpoints.add(
                    EndPoint(
                        "$url/:id/${it.name.toLowerCase()}", null, "id", "List<$name>", "get",
                        "handleGet${it.name}For$name", false
                    )
                )
            }
            return endpoints
        }
        private fun createEntity(aClass: Class<*>): Entity {
            return Entity(
                aClass.simpleName, getParents(aClass),
                isVerticleAbstract(aClass),
                shouldGenerateCrud(aClass),
                messagesDescriptor.createFields(aClass), listOf()
            )
        }
        private fun isVerticleAbstract(aClass: Class<*>): Boolean {
            return aClass.getDeclaredAnnotation(AbstractImplementation::class.java) != null
        }
        private fun getParents(aClass: Class<*>): List {
            return aClass.getDeclaredAnnotation(ChildOf::class.java)?.parents?.map { it.simpleName }?.requireNoNulls()
                ?: listOf()
        }
        private fun shouldGenerateCrud(aClass: Class<*>): CrudFeatures {
            val listAnno = aClass.getDeclaredAnnotation(GenerateList::class.java)
            val createAnno = aClass.getDeclaredAnnotation(GenerateCreate::class.java)
            val getAnno = aClass.getDeclaredAnnotation(GenerateGetById::class.java)
            val updateAnno = aClass.getDeclaredAnnotation(GenerateUpdate::class.java)
            val deleteAnno = aClass.getDeclaredAnnotation(GenerateDelete::class.java)
            return CrudFeatures(
                list = listAnno != null,
                create = createAnno != null,
                update = updateAnno != null,
                delete = deleteAnno != null,
                get = getAnno != null
            )
        }
    }
    class MessagesCreator(private val typeMapper: TypeMapper, private val frontendAnnotationsPackageName: String) {
        fun createMessages(packageName: String): List {
            val reflections = Reflections(packageName, SubTypesScanner(false))
            return reflections.allTypes.map { Class.forName(it) }.map { createMessages(it) }
        }
        private fun createMessages(aClass: Class<*>): Message {
            return Message(aClass.simpleName, createFields(aClass))
        }
        fun createFields(aClass: Class<*>): List {
            return ReflectionUtils.getAllFields(aClass).map { createField(it) }
        }
        private fun createField(field: java.lang.reflect.Field): Field {
            val annotations = field.declaredAnnotations
            return Field(
                field.name, typeMapper.map(field.type),
                createConstraints(annotations),
                annotations.map { anno -> anno::annotationClass.get() }.contains(FindBy::class)
            )
        }
        private fun createConstraints(annotations: Array): List {
            return annotations.filter { it.toString().startsWith("@$frontendAnnotationsPackageName") }
        }
    }
    class RoutersCreator(private val typeMapper: TypeMapper, private val endpointsPackage:String ) {
        fun createRouters(): List {
            val reflections = Reflections(endpointsPackage, SubTypesScanner(false))
            return reflections.allTypes.map {
                createRouter(
                    Class.forName(
                        it
                    )
                )
            }
        }
        private fun createRouter(aClass: Class<*>): Router {
            return Router(aClass.simpleName, getUrl(aClass),
                ReflectionUtils.getAllMethods(aClass).map {
                    createEndpoint(it)
                })
        }
        private fun getUrl(aClass: Class<*>): String {
            return aClass.getAnnotation(EndpointController::class.java).url
        }
        private fun getEndPointMethodName(declaredAnnotation: Endpoint?): String {
            val httpMethodName = declaredAnnotation?.method
            return (httpMethodName ?: HttpMethodName.GET).name.toLowerCase()
        }
        private fun getParamName(declaredAnnotation: Endpoint?): String {
            val paramName = declaredAnnotation?.param
            return (paramName ?: "id")
        }
        private fun createEndpoint(method: Method): EndPoint {
            val types = method.parameterTypes
            val declaredAnnotation: Endpoint? = method.getDeclaredAnnotation(Endpoint::class.java)
            val methodName = getEndPointMethodName(declaredAnnotation)
            var url = method.name
            var input: String? = null
            var param: String? = null
            val hasInput = types.isNotEmpty()
            val handlerName = "$methodName${StringUtils.capitalize(url)}"
            if (hasInput) {
                val inputType = types[0]
                val inputTypeName = typeMapper.map(inputType)
                val createUrlParameterName = inputType == java.lang.String::class.java
                if (createUrlParameterName) {
                    param = getParamName(declaredAnnotation)
                    url += "/:$param"
                } else {
                    input = simpleName(inputTypeName)
                }
            }
            return EndPoint(
                url, input, param, method.returnType.toString(),
                methodName, handlerName, isHandlerAsync(method)
            )
        }
        private fun isHandlerAsync(method: Method): Boolean {
            val declaredAnnotation: AsyncHandler? = method.getDeclaredAnnotation(AsyncHandler::class.java)
            return declaredAnnotation != null
        }
        private fun simpleName(name: String): String {
            val index = name.lastIndexOf(".")
            return if (index >= 0) name.substring(index + 1) else name
        }
    }
    


    Well, there are also “main” classes that receive arguments - which packages should be reflected on, which Velocity templates to use, etc.
    They are not so interests, you can look at everything in the repository: Source code

    In the frontend-generated and backend projects, we do similar things:
    1. dependency on api at compile time
    2. dependency on codegen at build stage
    3. Generation templates are in the buildSrc directory into which gradle puts the files and code that are needed at the build stage, but not at the compilation or runtime stage. Those. we can change the generation pattern without recompiling the codegen
    4 project .frontend-generated compiles the generated Typescript and publishes it to the npm package repository
    5. In the backend , routers are generated that inherit from a non-generated abstract router that knows how to handle different types of requests. Abstract Verticles are also generated that must be inherited with the implementation of the business logic itself. In addition, all sorts of little things are generated that I as a programmer do not want to think about - registering codecs and address constants in the event bass.
    Source code frontend-generated and backend.

    In frontend-generatedyou need to pay attention to the plugin that publishes the generated sources in the npm repository. For this to work, you need to put the IP of your repository in build.gradle and put your authentication token in .npmrc The

    generated classes look like this:
    import {
        IsString,
        MaxLength,
    IsDate,
    IsArray,
    } from 'class-validator';
    import { Type } from 'class-transformer';
    // Entity(parents=[], abstractVerticle=false, crudFeatures=CrudFeatures(list=true, create=true, update=true, delete=true, get=true), children=[])
    export class Chapter {
        // Field(name=bookId, type=number, validators=[], findBy=false)
         bookId!: number;
          @IsString()
          @MaxLength(128)
        // Field(name=name, type=string, validators=[@com.valapay.test.annotations.frontend.IsString(), @com.valapay.test.annotations.frontend.MaxLength(len=128)], findBy=false)
         name!: string;
        // Field(name=id, type=number, validators=[], findBy=false)
         id!: number;
        // Field(name=page, type=number, validators=[], findBy=false)
         page!: number;
    }
    


    Pay attention to the class-validator annotations tc.

    In the backend project, repositories for Spring Data JPA are also generated, it is possible to say that the message processing project in Verticle is blocking (and run through Vertx.executeBlocking) or asynchronous (with coroutines), it is possible to say that the Verticle generated for Entity is abstract and then there is a possibility override hooks that are called before and after calling the generated methods. Verticles deployment is automatic on the interface of spring beans - well, in short, a lot of goodies.

    And all this is easy to expand - for example, hanging a list of roles on Endpoints and generating a check of the role of the logged-in user when calling the endpoint and much more - that’s enough for that.

    It is also easy to generate not Vertx, not Spring, but something else - even akka-http, it is enough to only change the templates in the backend project.

    Another possible development direction is to generate more front-end.

    All source code is here .

    Thanks to Ildar from the front-end for help in creating the generation in our project and in writing the article

    Also popular now: