June 11, 2019 at 11:20Into the future with Jenkins & Oracle APEX service integration
In the process of managing large amounts of data and their visualization, we are increasingly faced with the need to introduce a single tool for access to functional management in the repository. Moreover, this tool should be equally convenient and understandable to all employees of the data management division: administrators, analysts, developers and testers.
My name is Vitaliy, I am responsible in Rostelecom for the direction of testing automation and implementation of DevOps processes. In this article, I will talk about an interesting solution to this problem in a large data warehouse of Rostelecom.
And this is how we came to such a set of tools for solving this problem.
For starters, a little background. Earlier, I wrote that we wrote our own module in Python to automate the process of installing objects in a data warehouse. It is controlled by Jenkins and allows you to start the necessary functionality either manually by a button with the input of the necessary startup parameters, or fully automatically according to a schedule without user intervention. Also, our company has already implemented a number of applications written using Oracle APEX.
Oracle APEX - in earlier versions of the application was called HTML DB. Using this tool, using only a browser and having programming experience in languages such as PL / SQL and JavaScript, you can develop fast, scalable and secure web applications that can be easily deployed on any circuit for development, testing and subsequent implementation in production .
To build a data entry form, there is no need to program the interface in a complex way. The application has many ready-made templates that can be reused to develop your own solution. End users also gain access to the application through a browser, thereby eliminating the need to install the application on a computer. And with all of the above pluses, the metadata (information about the data used) in our repository is in the Oracle database, so we could not ignore this tool. Oracle REST Data Services (ORDS)
APEX Application Architecture Example is a data service that replaces Oracle HTTP server and mod_plsql based on Java EE
. It provides interaction with objects through RESTfull.
Many, probably, already know what kind of beast it is, but for those who are just looking at this tool, I will tell you what it is and how we use it in our repository.
Jenkins is a continuous integration tool, most often used for software development and application deployment at different stages of the development process. In simple terms, Jenkins provides a process automation environment. Due to its flexibility and a large set of plug-ins, it makes it possible to integrate into any application or technology and supports working with different version control systems.
In our department of data management with its help, CI / CD processes with the run of automated tests are implemented. And Jenkins manages several self-written modules, thanks to which many manual processes are automated. Jenkins web-client interface example, clickable
Having implemented many tasks and automating them, we encountered a problem: not only DevOps engineers and testing specialists will have to manage these processes, but also people who do not encounter such tools, but are involved in the development process. End users were all internal employees of the department. The Jenkins client interface, in my opinion, is quite simple and convenient, but I look at it through the eyes of DevOps, and not everyone can look at it the same way. Since a number of tasks we had to run on the button by employees, it became necessary to come up with an interface that would be friendlier for the user. In fact, there is a plugin for Jenkins called Blue Ocean , which allows you to change the UI representation of the tool.
An example of the web-client interface Jenkins (Blue Ocean plugin), clickable
Our plug-in could not solve our problem, but as an alternative, if the standard interface does not work, it can be reconfigured. Most often, it becomes necessary to integrate the module into Jenkins, and not vice versa. This is the interest of the problem being solved. At the time of its solution, as I wrote, we already had a number of applications written using Oracle APEX. An example of the interface of one of the APEX applications, clickable Having checked that its interface is friendly enough and it is possible to manage it, and all the parameters for launching the necessary tasks in Jenkins are stored in the Oracle database, the idea came up to run a number of Jenkins tasks from APEX.
It took quite a bit of time. The interaction between the applications was implemented according to the architectural principle of building service-oriented systems such as REST. REST architecture implies the rules for the interaction of components of a distributed application on the network. APEX-application allows you to use this style and provides a ready-made template for the formation of the process of sending HTTP / HTTPs-requests of type REST during application development. As a result, we quickly lifted the application to launch such tasks, the data for launch began to be pulled directly from the database with the possibility of their selection, which eliminated the possibility of an error in entering startup parameters. The transmission of parameters for running tasks in Jenkins is carried out by means of a POST request, in the body of which lies JSON with the necessary parameters.
REST call form when developing an APEX application, clickable
JSON example:
An example of a ready-made APEX application for invoking Jenkins processes, clickable.
The result of clicking the “Run process” button will transfer parameters to the Jenkins task and then launch it. Displayed logs of successful and unsuccessful launches, which is directly returned from the console of the Jenkins task execution process. The task itself can contain any automated process.
The integration of application data in our department showed an excellent result. There is no longer a need to force people who should not work directly with Jenkins to understand the toolkit of the application. We were able to differentiate and leave control over the execution of tasks at the Jenkins level using a custom rights matrix so that users had the ability to build tasks, but did not have the ability to edit them.
Since this experiment was recognized as successful, development ideas appeared in the form of writing our own open source frameworks to simplify interaction with Jenkins and other applications that work with us. But this is a completely different story, which I will try to tell about in future articles.
For starters, a little background. Earlier, I wrote that we wrote our own module in Python to automate the process of installing objects in a data warehouse. It is controlled by Jenkins and allows you to start the necessary functionality either manually by a button with the input of the necessary startup parameters, or fully automatically according to a schedule without user intervention. Also, our company has already implemented a number of applications written using Oracle APEX.
What is Oracle Application Express (Oracle APEX)?
Oracle APEX - in earlier versions of the application was called HTML DB. Using this tool, using only a browser and having programming experience in languages such as PL / SQL and JavaScript, you can develop fast, scalable and secure web applications that can be easily deployed on any circuit for development, testing and subsequent implementation in production .
To build a data entry form, there is no need to program the interface in a complex way. The application has many ready-made templates that can be reused to develop your own solution. End users also gain access to the application through a browser, thereby eliminating the need to install the application on a computer. And with all of the above pluses, the metadata (information about the data used) in our repository is in the Oracle database, so we could not ignore this tool. Oracle REST Data Services (ORDS)
APEX Application Architecture Example is a data service that replaces Oracle HTTP server and mod_plsql based on Java EE
. It provides interaction with objects through RESTfull.
A bit about Jenkins
Many, probably, already know what kind of beast it is, but for those who are just looking at this tool, I will tell you what it is and how we use it in our repository.
Jenkins is a continuous integration tool, most often used for software development and application deployment at different stages of the development process. In simple terms, Jenkins provides a process automation environment. Due to its flexibility and a large set of plug-ins, it makes it possible to integrate into any application or technology and supports working with different version control systems.
In our department of data management with its help, CI / CD processes with the run of automated tests are implemented. And Jenkins manages several self-written modules, thanks to which many manual processes are automated. Jenkins web-client interface example, clickable
The problem of misunderstanding and its solution
Having implemented many tasks and automating them, we encountered a problem: not only DevOps engineers and testing specialists will have to manage these processes, but also people who do not encounter such tools, but are involved in the development process. End users were all internal employees of the department. The Jenkins client interface, in my opinion, is quite simple and convenient, but I look at it through the eyes of DevOps, and not everyone can look at it the same way. Since a number of tasks we had to run on the button by employees, it became necessary to come up with an interface that would be friendlier for the user. In fact, there is a plugin for Jenkins called Blue Ocean , which allows you to change the UI representation of the tool.
An example of the web-client interface Jenkins (Blue Ocean plugin), clickable
Our plug-in could not solve our problem, but as an alternative, if the standard interface does not work, it can be reconfigured. Most often, it becomes necessary to integrate the module into Jenkins, and not vice versa. This is the interest of the problem being solved. At the time of its solution, as I wrote, we already had a number of applications written using Oracle APEX. An example of the interface of one of the APEX applications, clickable Having checked that its interface is friendly enough and it is possible to manage it, and all the parameters for launching the necessary tasks in Jenkins are stored in the Oracle database, the idea came up to run a number of Jenkins tasks from APEX.
It took quite a bit of time. The interaction between the applications was implemented according to the architectural principle of building service-oriented systems such as REST. REST architecture implies the rules for the interaction of components of a distributed application on the network. APEX-application allows you to use this style and provides a ready-made template for the formation of the process of sending HTTP / HTTPs-requests of type REST during application development. As a result, we quickly lifted the application to launch such tasks, the data for launch began to be pulled directly from the database with the possibility of their selection, which eliminated the possibility of an error in entering startup parameters. The transmission of parameters for running tasks in Jenkins is carried out by means of a POST request, in the body of which lies JSON with the necessary parameters.
REST call form when developing an APEX application, clickable
JSON example:
{"parameter":[{"name":"SERVER","labels":"master"},{"name":"INSTANCE","value":"DEV"},{"name":"SRC_CODE_START","value":"SRC_ID"},{"name":"SRC_CODE_END","value":"SRC_ID"},{"name":"MODEL_ID","value":"MODEL_ID"},{"name":"SAVE_DATA","value":"0"}],"statusCode":"303","redirectTo":"."}
An example of a ready-made APEX application for invoking Jenkins processes, clickable.
The result of clicking the “Run process” button will transfer parameters to the Jenkins task and then launch it. Displayed logs of successful and unsuccessful launches, which is directly returned from the console of the Jenkins task execution process. The task itself can contain any automated process.
Eventually
The integration of application data in our department showed an excellent result. There is no longer a need to force people who should not work directly with Jenkins to understand the toolkit of the application. We were able to differentiate and leave control over the execution of tasks at the Jenkins level using a custom rights matrix so that users had the ability to build tasks, but did not have the ability to edit them.
Since this experiment was recognized as successful, development ideas appeared in the form of writing our own open source frameworks to simplify interaction with Jenkins and other applications that work with us. But this is a completely different story, which I will try to tell about in future articles.