Changing global settings on local command servers in the infrastructure using Gitlab CI and Ansible [Concept]


    This article describes the ability / idea / concept of changing global settings on local command servers in a large infrastructure using Gitlab CI and Ansible.


    Suppose you have 20 development teams and 1 admin / DevOps team. How to change admin passwords on all servers? How to add Enterprise root certificate to all servers? Etc.


    What problem solves?


    Typical situation:
    There are global administrators / DevOps.
    There are global settings (NTP, DNS, Proxy, etc.)
    There are local development teams: TeamA, TeamB, TeamC, TeamD, etc.
    There are developers who can only go to the servers of their team.
    How to add / update global administrators, global settings?



    The task is complicated by the fact that global settings are stored separately from local settings of commands in private repositories.


    If you have a few servers in the whole company, then you can run Ansible in a simple mode - start updating global administrators, global settings on all servers at once.


    For large installations, companies typically use Puppet, Chef.


    Concept


    To change global settings on local command servers in a large infrastructure, I suggest the git submodules mechanism.


    In the repository with local settings, git submodules with global settings are used.


    Below is a schematic diagram of connecting a private repository with global settings to local command repositories.



    You can use the concept of updating global settings using git submodule in infrastructure with Puppet, Chef, Salt, but the article gives an example with Ansible.


    For example, install tomcat, mysql, nginx and apply global settings to them in a team called team.


    In gitlab there is a common group, which contains common settings.


    The common group has a base-bootstrap project that contains administrators, sysctl settings, etc.


    Usually the company has several development departments - more often they are called teams.


    In gitlab, create a team group (you will have your own team name).


    In the team group we create projects: application, database, loadbalancer.


    Screenshot from base-bootstrap, application, database, loadbalancer:



    The base-bootstrap repository is enabled as a git submodule in the application, database, loadbalancer repository.


    At each commit in the application, database repository, loadbalancer starts the update of the base-bootstrap submodule.


    After that, ansible-playbook from base-bootstrap and ansible-playbook from base-bootstrap are applied to the application, database, loadbalancer servers.



    That is, if you add a new administrator to the base-bootstrap or change the system settings in the base-bootstrap, then with the application, database, loadbalancer commit, the new settings from the base-bootstrap will be applied in application, database, loadbalancer.


    Training


    It is necessary to read the articles about Ansible for beginners:


    # Ansible where to start


    # Ansible Handbook


    You should have gitlab and gitlab-runner deployed with the docker installed.


    The docker executor is used here as an example - you can use the Shell executor.


    How to deploy gitlab and gitlab-runner:


    # Article about Gitlab-CI from Southbridge


    # Continuous integration and deployment of Docker in GitLab CI


    You must have 3 servers (for example, on ubuntu): application, database, loadbalancer.


    Application, database, loadbalancer are common names.


    All examples can be expanded, improved, use other software - the article shows a general principle.


    How to implement


    The repository and all the code for the test can be taken from here: https://github.com/patsevanton/ansible-gitlab-habr


    If you go to the server by login / password, then in the right group (in this case, team) create the userpassword variable (when you change, you must also change the variable in the code) and specify the password there (the password is used in the code)


    Do not forget to create on the target servers the user you need with sudo rights (the code uses the user user).


    For those who go to the server via SSH keys, you need to create in the team group a variable SSH_PRIVATE_KEY and add a private user key to it, which will be connected to the server.


    Here is a simple example of connecting to servers, so security issues are not considered in the article.


    In each repository (application, database, loadbalancer) create a git submodule:


    git submodule add git@gitlab.example.com:common/base-bootstrap.git
git submodule add git@gitlab.example.com:team/team-users.git

    Submodule is needed to access the shared private repository.


    In our case, this is a repository with common base-bootstrap settings and a repository of team-users team users.


    Where gitlab.example.com is your gitlab server.


    Then in .gitmodules we change the path to the repository to relative


    Example:


    [submodule "team-users"]
  path = team-users
  url = ../team-users.git
[submodule "base-bootstrap"]
  path = base-bootstrap
  url = ../../common/base-bootstrap.git

    In each repository in hosts, we change the IP for ours, in ansible.cfg, we change the remote_user for our user.


    If you have no commits in the last few hours / days of the repository, and you need to roll out new general changes to the server (for example, you need to add a new admin) - for such situations there is a ansible-pull.


    Configure the ansible-pull to download the common / base-bootstrap repository.


    To do this, add the deploy token repository.


    Go to the common / base-bootstrap repository and then go to settings / repository / Deploy Tokens.


    Create a token. The resulting username and password are written to base-bootstrap / vars / cron.yml.


    After checking that everything works correctly - I think it is worth changing the start time of the ansible-pull from "every 2 minutes" to the one that suits you.


    If ansible-pull is dropped, it means that the CI of this service will drop, which starts at every commit to this service repository (say, the service is called application)


    Check


    Creating a new administrator.


    Try adding a new admin to https://github.com/patsevanton/ansible-gitlab-habr/blob/master/commons/base-bootstrap/vars/users.yml


    Change sysctl


    Try adding / changing sysctl settings at https://github.com/patsevanton/ansible-gitlab-habr/blob/master/commons/base-bootstrap/vars/sysctl.yml


    Add entries to cron


    Try adding cron entries to https://github.com/patsevanton/ansible-gitlab-habr/blob/master/commons/base-bootstrap/vars/cron.yml


    Expansion or installation of your applications


    First you need to find the role to install your applications.


    Go to https://galaxy.ansible.com/ and find the role to install your application.


    Try installing your application using the role from the console to your servers. Usually in the descriptions of all the roles there is an instruction.


    For example, try installing java next to tomcat. First install the geerlingguy.java role


    ansible-galaxy install geerlingguy.java

    Create standard ansible.cfg same as repositories.


    Create inventory:


    [java]
java ansible_host=IP-сервера

    Create a playbook java.yml


    - hosts: java
  become: yes
  vars_files:
    - vars/main.yml
  roles:
    - { role: geerlingguy.java }

    Run ansible-playbook java.yml


    If everything went successfully add to the desired project (in this case, the application)


    The role of geerlingguy.java is added in after the role of robertdebock.tomcat https://github.com/patsevanton/ansible-gitlab-habr/blob/master/team/application/tomcat-app.yml#L11


    The same with all other applications that you need to install on the server.


    Playbook Testing and Security


    To simplify the article, the issue of storing passwords and testing is not affected.


    There are articles on testing a playbook:
    # Ansible: testing playbooks (part 1)


    # Testing and continuous integration for Ansible roles with Molecule and Jenkins


    Answers on questions


    1) Mentat: And why is it still not like it is written in the ansible dock, with environments? From the first reading it looks like an attempt to reinvent it all over again. It’s quite convenient to use all of this types. Ansible-playbook -i env / teamA personalAPlaybook.yml
    Answer: This scheme allows you to change global settings. What is described in the question is a change in the local settings of the commands.


    PS Perhaps the same functionality is implemented in Ansible Tower. But I can’t say anything about this - I didn’t work with Ansible Tower.

    Tags:

    Also popular now: