December 13, 2012 at 12:03The key to all doors in continuous integration - rundeck
With a large number of servers and virtual machines and even more code in a constant deploy, there will inevitably be problems with the administration of this huge economy. There are many tools to organize continuous integration. Our list definitely already has GIT, Jenkins, Chef, Proxmox, Graylog2. Today we will talk about another convenient tool for automating routine tasks using scripts - rundeck . This article is not a detailed manual with examples of configs, but rather reflections on the topic.
Rundeck is an open source project that will allow the system administrator to organize a script server, automating routine and template work. In fact, this is the key to all doors. It allows you to execute scripts or shell commands on any number of nodes, monitor their execution and tell the administrator in detail about each action, successful or not. Rundeck combines all the tools already used and allows you to organize a transparent interaction between them.
- distributed execution of commands and scripts with parameters,
- support for several mechanisms for interaction with a node (ssh by default) and the ability to write your own in the form of plug-ins,
- a script can be described in several steps,
- a graphical control interface and console utilities for configuration,
- separation of rights access, integration with LDAP / AD,
- history and audit of scripts and commands,
- integration with other continuous integration tools - jenkins, chef,
- API,
- the ability to take all parameters and settings by url from a remote host.
Is a server and it is java. Here, all the logic, authentication and access rights for users, console management utilities and a web-based management interface (though not very convenient), such
as scripts, audit results, and all logs are stored in a database,
use ssh for keys to interact with nodes. In this case, a reverse ssh node-server connection is not required.
It works almost everywhere:
- linux,
- Windows XP, Server and later,
- Mac OS X 10.4 and later.
Installation and configuration is trivial, all configs are stored in files.
Our use case, integration with chef. Problems and solutions.
A little lyrical digression about chef . Recipes, nodes, environment - all this works in theory with almost no problems. Problems begin in real production. The connectivity of the system components is unexpectedly high, some modules are very dependent on others. In a geo-cluster and distributed data centers, many factors must be considered. Network speed, speed of access to a node from different geographic points, time zones. For large audiences, split deploy can be used when a new feature is shown only to certain regions and only at a certain time.
In such realities, chef already ceases to save and harms more. There are too many recipes. Places where you can make mistakes more and more. There are more options, environments and flags for recipes than recipes. We have to start long and lengthy tables, where the scheme of our military operations will be described with a multi-colored felt-tip pen. Where, where and at what time did we deploy and why. Once we nevertheless split the wrong way, which led to the understanding that chef also needs to be automated and that it also needs to be managed.
The recipes had a lot of logic to overcome the high connectivity of the components. The node turns off from the upstream, prepares its code and begins to deploy. At some specific step, it is necessary to change the state of another module, for example, to reindex data for a new code. After reindexing, we immediately extinguish another node and prepare it for split testing. At this time, the first node finishes its deployment and passes the final testing, after which it is included in the upstream. In this case, a database should already be ready, where a million transactions have something updated at the same time. All this must be done almost simultaneously, while almost synchronously. Chef seems to be able to, but it is inconvenient and you have to use too much logic in the recipes. Share signals, common attributes,
What gives us rundeck.
1. We create the project
2. Import there all notes from chef with all recipes. There is a ready-made tool for this .
3. We create scenarios that will do everything ourselves and make sure that the connectivity did not break during the deployment. Turn on, turn off, synchronize, kick the check and report to the administrator about each step.
4. Profit.
To do this, you need to configure key authentication. Understand what and how an unprivileged user should do, configure sudo. Or shove it in a chroot. Well, write a bunch of magic scripts in any language. Perl, python, bash, php))) You can also enable the script to stop execution after an unsuccessful step. Scripts can be written as you like. It all depends on the imagination and the challenges facing you.
Rundeck can work in two modes. Tasks can be performed either in parallel on all nodes at once, or sequentially, finished with one - go to the other.
Node-oriented
1. NodeA step # 1
2. "step # 2
3." step # 3
4. NodeB step # 1
5. "step # 2
6." step # 3
Step oriented
1. NodeA step # 1
2. NodeB "
3. NodeA step # 2
4. NodeB"
5. NodeA step # 1
6. NodeB "
Rundeck can execute commands and scripts with arguments, and they, in turn, can be taken from the remote url, like the rest of the settings. You can also use the API Manage only those things that are needed without opening the webUI, and it’s not very convenient here.
A big plus of this solution is that the state of the system as a whole and individually at any time is monitored from one point. And it can be forcibly changed, also from one point. Without the need to ssh-go to megatons of servers.
Of the minuses - one more component from the series monitor monitoring monitoring and deploy deployment deploy system deployment. Complicating is almost always bad. You need to clearly understand the layout of the entire system and components individually. Otherwise, there is a chance to script such miracles that even dismissal would be an insufficient measure of punishment. You must also clearly and clearly build a security scheme. And allow the user only what needs to be allowed. The key to all the doors is convenient, but it wouldn’t happen that “we are super scientists and have come up with a vigorous bomb. But if she falls into the wrong ruuuki ... "
Links
- Chef
- Rundeck
Rundeck is an open source project that will allow the system administrator to organize a script server, automating routine and template work. In fact, this is the key to all doors. It allows you to execute scripts or shell commands on any number of nodes, monitor their execution and tell the administrator in detail about each action, successful or not. Rundeck combines all the tools already used and allows you to organize a transparent interaction between them.
Features:
- distributed execution of commands and scripts with parameters,
- support for several mechanisms for interaction with a node (ssh by default) and the ability to write your own in the form of plug-ins,
- a script can be described in several steps,
- a graphical control interface and console utilities for configuration,
- separation of rights access, integration with LDAP / AD,
- history and audit of scripts and commands,
- integration with other continuous integration tools - jenkins, chef,
- API,
- the ability to take all parameters and settings by url from a remote host.
Architecture:
Is a server and it is java. Here, all the logic, authentication and access rights for users, console management utilities and a web-based management interface (though not very convenient), such
as scripts, audit results, and all logs are stored in a database,
use ssh for keys to interact with nodes. In this case, a reverse ssh node-server connection is not required.
It works almost everywhere:
- linux, - Windows XP, Server and later,
- Mac OS X 10.4 and later.
Installation and configuration is trivial, all configs are stored in files.
Our use case, integration with chef. Problems and solutions.
A little lyrical digression about chef . Recipes, nodes, environment - all this works in theory with almost no problems. Problems begin in real production. The connectivity of the system components is unexpectedly high, some modules are very dependent on others. In a geo-cluster and distributed data centers, many factors must be considered. Network speed, speed of access to a node from different geographic points, time zones. For large audiences, split deploy can be used when a new feature is shown only to certain regions and only at a certain time.
In such realities, chef already ceases to save and harms more. There are too many recipes. Places where you can make mistakes more and more. There are more options, environments and flags for recipes than recipes. We have to start long and lengthy tables, where the scheme of our military operations will be described with a multi-colored felt-tip pen. Where, where and at what time did we deploy and why. Once we nevertheless split the wrong way, which led to the understanding that chef also needs to be automated and that it also needs to be managed.
The recipes had a lot of logic to overcome the high connectivity of the components. The node turns off from the upstream, prepares its code and begins to deploy. At some specific step, it is necessary to change the state of another module, for example, to reindex data for a new code. After reindexing, we immediately extinguish another node and prepare it for split testing. At this time, the first node finishes its deployment and passes the final testing, after which it is included in the upstream. In this case, a database should already be ready, where a million transactions have something updated at the same time. All this must be done almost simultaneously, while almost synchronously. Chef seems to be able to, but it is inconvenient and you have to use too much logic in the recipes. Share signals, common attributes,
What gives us rundeck.
1. We create the project 2. Import there all notes from chef with all recipes. There is a ready-made tool for this .
3. We create scenarios that will do everything ourselves and make sure that the connectivity did not break during the deployment. Turn on, turn off, synchronize, kick the check and report to the administrator about each step.
4. Profit.
To do this, you need to configure key authentication. Understand what and how an unprivileged user should do, configure sudo. Or shove it in a chroot. Well, write a bunch of magic scripts in any language. Perl, python, bash, php))) You can also enable the script to stop execution after an unsuccessful step. Scripts can be written as you like. It all depends on the imagination and the challenges facing you.
Rundeck can work in two modes. Tasks can be performed either in parallel on all nodes at once, or sequentially, finished with one - go to the other.
Node-oriented
1. NodeA step # 1
2. "step # 2
3." step # 3
4. NodeB step # 1
5. "step # 2
6." step # 3
Step oriented
1. NodeA step # 1
2. NodeB "
3. NodeA step # 2
4. NodeB"
5. NodeA step # 1
6. NodeB "
Rundeck can execute commands and scripts with arguments, and they, in turn, can be taken from the remote url, like the rest of the settings. You can also use the API Manage only those things that are needed without opening the webUI, and it’s not very convenient here.
A big plus of this solution is that the state of the system as a whole and individually at any time is monitored from one point. And it can be forcibly changed, also from one point. Without the need to ssh-go to megatons of servers.
Of the minuses - one more component from the series monitor monitoring monitoring and deploy deployment deploy system deployment. Complicating is almost always bad. You need to clearly understand the layout of the entire system and components individually. Otherwise, there is a chance to script such miracles that even dismissal would be an insufficient measure of punishment. You must also clearly and clearly build a security scheme. And allow the user only what needs to be allowed. The key to all the doors is convenient, but it wouldn’t happen that “we are super scientists and have come up with a vigorous bomb. But if she falls into the wrong ruuuki ... "
Links
- Chef
- Rundeck