DevOps in development: web application code automation

Good day, dear Habrazhiteli!

Today DevOps is on the wave of success. In almost any conference dedicated to automation, you can hear from the speaker saying “we implemented DevOps here and there, applied this and that, it became much easier to conduct projects, etc., etc.”. And it is commendable. But, as a rule, the implementation of DevOps in many companies ends at the stage of automation of IT Operations, and very few people talk about implementing DevOps directly in the development process itself.

I would like to correct this small misunderstanding. DevOps can come into development through the formalization of the code base, for example, when writing a GUI for the REST API.

In this article, I would like to share with you the solution to the non-standard case that our company encountered - we were able to automate the formation of the web application interface. I will tell you about how we came to this task and what we used to solve it. We do not believe that our approach is the only true one, but we really like it.

I hope this material will be interesting and useful to you.

Well, let's get started!

Background


This story began about a year ago: it was a beautiful summer day and our development department was creating the next web application. On the agenda was the task of introducing a new feature into the application - it was necessary to add the ability to create custom hooks.

The process of adding new features on old architecture

At that time, the architecture of our web application was built in such a way that in order to implement a new feature, we needed to do the following:

  1. At the back-end: create a model for a new entity (hooks), describe the fields of this model, describe the entire logic of actions that the model can perform, etc.
  2. At the front-end: create a view class that corresponds to the new model in the API, manually describe all the fields that this model has, add all types of actions that this view can run, etc.

It turns out that we simultaneously at once in two places, it was necessary to make very similar changes in the code, one way or another, "duplicating" each other. And this, as you know, is not good, because with further changes, developers would need to make changes in the same place in two places at the same time.

Suppose we need to change the type of the “name” field from “string” to “textarea”. To do this, we will need to make this amendment in the model code on the server, and then make similar changes to the presentation code on the client.

Is it too complicated?

Previously, we put up with this fact, since many applications were not very large and there was a place to “duplicate” the code on the server and on the client. But on that very summer day, before the introduction of the new feature, something clicked inside us, and we realized that we couldn’t work like that anymore. The current approach was very unreasonable and required a lot of time and labor. In addition, “duplication” of code on the back-end and front-end could lead to unexpected bugs in the future: developers could make changes on the server and forget to make similar changes on the client, and then everything would not go well according to plan.

How to avoid code duplication? Search for a solution


We began to wonder how we can optimize the process of introducing new features.

We asked ourselves the question: “Can we immediately avoid duplicating changes in the model’s representation at the front-end'e, after any change in its structure at the back-end'e?”

We thought and answered: “No, we can’t” .

Then we asked ourselves another question: “Okay, what then is the reason for such code duplication?”

And it dawned on us: the problem, in fact, is that our front-end does not receive data about the current API structure. Front-end does not know anything about the models that exist in the API until we ourselves inform it about it.

And then we got the idea: what if we build the application architecture in such a way that:

  • Front-end received from the API not only model data, but also the structure of these models;
  • Front-end dynamically formed representations based on the structure of models;
  • Any change in the structure of the API was automatically displayed on the front-end.

Implementing a new feature will take much less time, because it will require changes only on the back-end side, and the front-end will automatically pick up everything and present it to the user properly.

The versatility of the new architecture


And then, we decided to think a little more broadly: is the new architecture suitable only for our current application, or can we use it somewhere else?

Features common to many web applications

Indeed, one way or another, almost all applications have part of a similar functionality:

  • almost all applications have users, and in this regard, it is necessary to have functionality associated with user registration and authorization;
  • almost all applications have several types of views: there is a view for viewing a list of objects of a model, there is a view for viewing a detailed record of a single, individual, model object;
  • almost all models have similar attributes in type: string data, numbers, etc., and in this regard, you need to be able to work with them both on the back-end and on the front-end.

And since our company often develops custom web applications, we thought: why do we need to reinvent the wheel every time and develop similar functionality every time from scratch, if we can write a framework once that describes all the basic, common for many applications, things, and then, creating a new project, use ready-made developments as dependencies, and if necessary, declaratively change them in a new project.

Thus, in the course of a long discussion, we had the idea of ​​creating VSTUtils - a framework that would:

  1. It contained the basic functionality, most similar to most applications;
  2. Allowed to generate front-end on the fly, based on the structure of the API.

How to make friends back-end and front-end?


Well then, we have to do, we thought. We already had some back-end, some front-end too, but neither the server nor the client had a tool that could report or receive data on the structure of the API.

In the search for a solution to this problem, our eye fell on the OpenAPI specification , which, based on the description of the models and the relationships between them, generates a huge JSON containing all this information.

And we thought that, in theory, when initializing the application on the client, the front-end can receive this JSON from the API and build all the necessary views on its basis. It remains only to teach our front-end to do all this.

And after some time we taught him.

Version 1.0 - what came out of it


The architecture of the VSTUtils framework of the first versions consisted of 3 conditional parts and looked something like this:

  1. Back end:
    • Django and Python are all model related logic. Based on the base Django Model, we have created several classes of core VSTUtils models. All actions that these models can perform we implemented using Python;
    • Django REST Framework - REST API generation. Based on the description of the models, a REST API is formed, thanks to which the server and client communicate;
  2. Interlayer between back-end and front-end:
    • OpenAPI - JSON generation with a description of the API structure. After all models have been described on the back-end, views are created for them. Adding each of the views introduces the necessary information into the resulting JSON:
      JSON Example - OpenAPI Schema
      {
          // объект, хранящий в себе пары (ключ, значение),
          // где ключ - имя модели,
          // значение - объект с описанием полей модели.
          definitions: {
              // описание структуры модели Hook.
              Hook: {
                  // объект, хранящий в себе пары (ключ, зачение),
                  // где ключ - имя поля модели,
                  // значение - объект с описанием свойств данного поля (заголовок, тип поля и т.д.).
                  properties: {
                      id: {
                          title: "Id",
                              type: "integer",
                              readOnly: true,
                      },
                      name: {
                          title: "Name",
                              type: "string",
                              minLength:1,
                              maxLength: 512,
                      },
                      type: {
                          title: "Type",
                              type: "string",
                      enum: ["HTTP","SCRIPT"],
                      },
                      when: {
                          title: "When",
                              type: "string",
                      enum: ["on_object_add","on_object_upd","on_object_del"],
                      },
                      enable: {
                          title:"Enable",
                              type:"boolean",
                      },
                      recipients: {
                          title: "Recipients",
                              type: "string",
                              minLength: 1,
                      }
                  },
                  // массив, хранящий в себе имена полей, являющихся обязательными для заполнения.
                  required: ["type","recipients"],
              }
          },
          // объект, хранящий в себе пары (ключ, значение),
          // где ключ - путь предсталения (шаблонный URL),
          // значение - объект с описанием свойств представления.
          paths: {
              // описание структуры представлений по пути '/hook/'.
              '/hook/': {
                  // схема представления для get запроса по пути /hook/.
                  // схема представления, соответствующей странице просмотра списка объектов модели Hook.
                  get: {
                      operationId: "hook_list",
                          description: "Return all hooks.",
                          // массив, хранящий в себе объекты со свойствами фильтров, доступных для данного списка объектов.
                          parameters: [
                          {
                              name: "id",
                              in: "query",
                              description: "A unique integer value (or comma separated list) identifying this instance.",
                              required: false,
                              type: "string",
                          },
                          {
                              name: "name",
                              in: "query",
                              description: "A name string value (or comma separated list) of instance.",
                              required: false,
                              type: "string",
                          },
                          {
                              name: "type",
                              in: "query",
                              description: "Instance type.",
                              required: false,
                              type: "string",
                          },
                      ],
                          // объект, хранящий в себе пары (ключ, значение),
                          // где ключ - код ответа сервера;
                          // значение - схема ответа сервера.
                          responses: {
                          200: {
                              description: "Action accepted.",
                                  schema: {
                                  properties: {
                                      results: {
                                          type: "array",
                                              items: {
                                              // ссылка на модель, данные которой пришли в ответе от сервера.
                                              $ref: "#/definitions/Hook",
                                          },
                                      },
                                  },
                              },
                          },
                          400: {
                              description: "Validation error or some data error.",
                                  schema: {
                                  $ref: "#/definitions/Error",
                              },
                          },
                          401: {
                              // ...
                          },
                          403: {
                              // ...
                          },
                          404: {
                              // ...
                          },
                      },
                      tags: ["hook"],
                  },
                  // схема представления для post запроса по пути /hook/.
                  // схема представления, соответствующей странице создания нового объекта модели Hook.
                  post: {
                      operationId: "hook_add",
                          description: "Create a new hook.",
                          parameters: [
                          {
                              name: "data",
                              in: "body",
                              required: true,
                              schema: {
                                  $ref: "#/definitions/Hook",
                              },
                          },
                      ],
                          responses: {
                          201: {
                              description: "Action accepted.",
                                  schema: {
                                  $ref: "#/definitions/Hook",
                              },
                          },
                          400: {
                              description: "Validation error or some data error.",
                                  schema: {
                                  $ref: "#/definitions/Error",
                              },
                          },
                          401: {
                              // ...
                          },
                          403: {
                              // ...
                          },
                          404: {
                              // ...
                          },
                      },
                      tags: ["hook"],
                  },
              }
          }
      }
  3. Front-end:
    • JavaScript is a mechanism that parses an OpenAPI scheme and generates views. This mechanism is launched once, when the application is initialized on the client. By sending a request to the API, it receives the requested JSON in response with a description of the API structure and, analyzing it, creates all the necessary JS objects containing the parameters of the model representations. This API request is quite heavy, so we cache it and request it again only when updating the application version;
    • JavaScript SPA libs - rendering views and routing between them. These libraries were written by one of our front-end developers. When a user accesses a particular page, the rendering engine draws the page based on the parameters stored in JS representation objects.

Thus, what we have: we have a back-end that describes all the logic associated with models. Then OpenAPI enters the game, which, based on the description of the models, generates JSON with a description of the API structure. Next, the baton is transmitted to the client, which, analyzing the generated OpenAPI JSON automatically generates a web interface.

Embedding features in the application on the new architecture - how it works


Remember the task of adding custom hooks? Here's how we would implement it in an application based on VSTUtils:

The process of adding new features to the new architecture

Now thanks to VSTUtils we do not need to write anything from scratch. Here's what we do to add the ability to create custom hooks:

  1. At the back-end: we take and inherit from the most suitable class in VSTUtils, add new functionality specific to the new model;
  2. At the front end:
    • if the view for this model is no different from the basic view of VSTUtils, then we do nothing, everything is automatically displayed properly;
    • if you need to somehow change the behavior of the view, using the signal mechanism, we declaratively expand or completely change the basic behavior of the view.

As a result, we got a pretty good solution, we achieved our goal, our front-end became auto-generated. The process of introducing new features into existing projects has noticeably accelerated: releases began to be released every 2 weeks, whereas previously we released releases every 2-3 months with a much smaller number of new features. I would like to note that the development team has remained the same, it was the new application architecture that gave us the fruits.

Version 1.0 - our hearts demand change


But, as you know, there is no limit to perfection, and VSTUtils was no exception.

Despite the fact that we were able to automate the formation of the front-end, the result was not the direct solution that we originally wanted.

The client-side application architecture was not thoroughly thought out, and it turned out not as flexible as it could be:

  • the process of introducing functional overloads was not always convenient;
  • OpenAPI parsing mechanism was not optimal;
  • rendering of representations and routing between them was carried out using self-written libraries, which also did not suit us for a number of reasons:
    • These libraries were not covered by tests;
    • there was no documentation for these libraries;
    • they didn’t have any community - in case of detection of bugs in them or the departure of the employee who wrote them, support for such code would be very difficult.

And since in our company we adhere to the DevOps approach and try to standardize and formalize our code as much as possible, in February of this year we decided to conduct a global refactoring of the VSTUtils front-end framework. We had several tasks:

  • to form not only presentation classes at the front-end, but also model classes - we realized that it would be more correct to separate data (and their structure) from their presentation. In addition, the presence of several abstractions in the form of a representation and a model would greatly facilitate the addition of overloads of the basic functionality in projects based on VSTUtils;
  • use a tested framework with a large community (Angular, React, Vue) for rendering and routing - this will allow us to give away all the headache with support for code related to rendering and routing inside our application.

Refactoring - choice of JS framework


Among the most popular JS frameworks: Angular, React, Vue, our choice fell on Vue because:

  • Vue's code base weighs less than React and Angular;

    Comparison Chart frameworks Gzipped size version
    FrameworkSize, kb
    Angular 2111
    Angular 2 + RX143
    Angular 1.4.551
    React 0.14.5 + React DOM40
    React 0.14.5 + React DOM + Redux42
    React 15.3.0 + React DOM43
    Vue 2.4.221
  • Vue's page rendering process takes less time than React and Angular;
    comparing page rendering speed with different javascript frameworks relative to pure javascript
  • The entry threshold in Vue is much lower than in React and Angular;
  • Natively understandable syntax of templates;
  • Elegant, detailed documentation available in several languages, including Russian;
  • A developed ecosystem that provides, in addition to the Vue core library, libraries for routing and for creating a reactive data warehouse.

Version 2.0 - the result of front-end refactoring


The process of global refactoring of the front-end of VSTUtils took about 4 months and this is what we ended up with: The

new fron-end VSTUtils architecture

front-end framework of VSTUtils still consists of two large blocks: the first is engaged in parsing the OpenAPI scheme, the second in rendering views and routing between them , but both of these blocks suffered a number of significant changes.

The mechanism that washes the OpenAPI scheme has been completely rewritten. The approach to parsing this scheme has changed. We tried to make the front-end architecture as similar as possible to the back-end architecture. Now on the client side we have not just a single abstraction in the form of representations, now we also have abstractions in the form of models and QuerySets:

  • objects of the Model class and its descendants are objects corresponding to the server-side abstractions of Django Models. Objects of this type contain data on the structure of the model (model name, model fields, etc.);
  • objects of the QuerySet class and its descendants are objects corresponding to the server-side Django QuerySets abstraction. Objects of this type contain methods that allow you to perform API requests (add, modify, receive, delete data of model objects);
  • objects of the View class - objects that store data about how to represent the model on a particular page, which template to use to “render” the page, which other representations of the models this page can link to, etc.

The unit responsible for rendering and routing has also changed significantly. We abandoned the self-written JS SPA libraries in favor of the Vue.js. We have developed our own Vue components that make up all the pages of our web application. Routing between views is done using the vue-router library, and we use vuex as the reactive storage of application state.

I would also like to note that on the front-end side the implementation of the Model, QuerySet and View classes does not depend on the means of rendering and routing, that is, if we suddenly want to switch from Vue to some other framework, for example, React or something new, then all we need to do is rewrite the Vue components to the components of the new framework, rewrite the router, the repository, and that’s all - the VSTUtils framework will work again. The implementation of the Model, QuerySet, and View classes will remain the same, since it does not depend on Vue.js. We believe that this is a very good help for possible future changes.

To summarize


Thus, the reluctance to write “duplicate” code resulted in the task of automating the formation of the front-end of a web application, which was solved by creating the VSTUtils framework. We managed to build the architecture of the web application so that the back-end and front-end harmoniously complement each other and any change in the API structure is automatically picked up and displayed properly on the client.

The benefits we have received from formalizing the architecture of the web application:

  • Releases of applications running on the basis of VSTUtils began to come out 2 times more often. This is due to the fact that now for introducing a new feature, often, we need to add code only on the back-end, the front-end will be automatically generated - this saves time;
  • Simplified updating the basic functionality. Since now all the basic functionality is assembled in one framework, in order to update some important dependencies or make an improvement in the basic functionality, we need to make changes in only one place - in the VSTUtils code base. When updating the version of VSTUtils in child projects, all innovations will automatically be picked up;
  • Finding new employees has become easier. Agree, it is much easier to find a developer for a formalized technology stack (Django, Vue) than to look for a person who agrees to work with an unknown recorder. Search results for developers who mentioned Django or Vue on HeadHunter in their CVs (across all regions):
    • Django - 3,454 resumes were found for 3,136 applicants;
    • Vue - 4,092 resumes were found for 3,747 job seekers.

The disadvantages of such a formalization of the architecture of a web application include the following:

  • Due to the parsing of the OpenAPI scheme, the initialization of the application on the client takes a little longer than before (about 20-30 milliseconds longer);
  • Unimportant search indexing. The fact is that at the moment we are not using server rendering in the framework of VSTUtils, and all the content of the application is formed in the final form already on the client. But for our projects, often high search results are not needed and for us it is not so critical.

On this my story comes to an end, thank you for your attention!

useful links



Also popular now: