Spacewalk for Christmas

Published on January 09, 2019

Spacewalk for Christmas

Hi, Habr!

Shortly before the arrival of Christmas, in the IT department it was decided to study Spacewalk, this is the Red Hat system, a free analogue of the Satellite, for centralized configuration management, system updates, and convenient support for the entire server fleet.

In view of the fact that the documentation available on the official website is rather scarce for additional comments of various kinds of troubleshooting, the task was to study the product for the gradual introduction of it first on test servers and subsequently on products.

The main idea of ​​the introduction of Spacewalk was not only centralization and simplification of control, but also that playful pens no one independently made updates on the servers of the new project, the benefit of the precedents already occurred.

After two weeks of work, all the knowledge I gained was entered into the internal analogue of Confluence, and a free weekend pushed me to write an article on Habr.

Before I begin, I would like to briefly highlight what was and was not affected, so as not to claim a complete manual for working with Spacewalk:

+ Installing and configuring the server / client
+ Setting up the system in the GUI
+ Trabbling installation / update packages, working with configurations
+ Errata (collection of information about critical updates, vulnerabilities, etc.)

- Proxy (need gone, after failure from HA)
- Cobbler / kickstart
- OpenSCAP

System requirements


Due to the fact that the entire infrastructure runs on VMWare, then the work was carried out on VMs running CentOS 7. The recommended system requirements from the developer are:

  • 4GB RAM
  • 6GB of free space for / var / satellite /
  • 12GB for db

I used:

  • 6GB RAM
  • 4 CPU (s)
  • 40GB HDD

I also advise you to disable SELinux and, if not, use firewalld. Or add http service to exceptions.

Note: at the end of the article there will be playbooks for Ansible, both for the client and for the server part, as well as bash scripts. With their help, you can deploy the entire infrastructure in a couple of minutes.

Installation


The installation itself is described both in official documentation and on several sites, but for the integrity of the article I will allow myself to mention this point here.

Spacewalk works with two DBMS: PostgreSQL and Oracle RDBMS. I have experience with the first, and I will use it now.

There are two installation options: via the automatic installer from Spacewalk, which installs and configures itself, and the database, however, on the same server, and installs manually, where you can install the DB and the application on different servers. I will consider both options, start with a separate installation.

PostgreSQL


yum install -y postgresql-server

You also need to connect PL / Tcl modules for PG:

yum install -y postgresql-pltcl
postgresql-setup initdb
systemctl start postgresql

Create a database, user, and connect the module:


su - postgres -c 'PGPASSWORD=verystrong; createdb spcwlkdb ; createlang plpgsql spcwlkdb ; createlang pltclu spcwlkdb ; yes $PGPASSWORD | createuser -P -sDR spcwlkuser'

To avoid problems with the connection, you should change /var/lib/pgsql/data/pg_hba.conf , adding the lines BEFORE the line all:

local spcwlkdb spcwlkuser md5
host  spcwlkdb spcwlkuser 127.0.0.1/8 md5
local spcwlkdb postgres   ident

Restart all:

systemctl restart postgresql

If you are going to install the application and the database on different servers, make sure that the postgresql-contrib package is installed on the database server .

Spacewalk


We connect repositories:

rpm -Uvh https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/spacewalk-2.8/epel-7-x86_64/00736372-spacewalk-repo/spacewalk-repo-2.8-11.el7.centos.noarch.rpm

Also connect epel:

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Java:

(cd /etc/yum.repos.d && curl -O https://copr.fedorainfracloud.org/coprs/g/spacewalkproject/java-packages/repo/epel-7/group_spacewalkproject-java-packages-epel-7.repo)

Now directly the installer for Spacewalk-postgres:

yum -y install spacewalk-setup-postgresql

Connect to our database:

spacewalk-setup-postgresql create --db spcwlkdb --user spcwlkuser --password verystrong

In that case, if you use the separate DB / application method, then you will need to add the --standalone flag and specify the IP address of the database server, also do not forget to open port 5432.

Note: I advise you to perform database tuning in accordance with your hardware for more productive work of the entire system.

Now install Spacewalk itself and run the installation:

yum -y install spacewalk-postgresql
spacewalk-setup --external-postgresql

Next will be asked a few questions, both for the SSL certificate and for the database. You can enter all values ​​manually, or you can use the --answer-file flag , and specify the path to the file with answers to automate the installation in the future:

admin-email = root@localhost
ssl-set-cnames = spcwlkserver
ssl-set-org = Unicorn
ssl-set-org-unit = EOH
ssl-set-city = Prague
ssl-set-state = HMP
ssl-set-country = CZ
ssl-password = verystrong
ssl-set-email = root@localhost
ssl-config-sslvhost = Y
db-backend=postgresql
db-name=spcwlkdb
db-user=spcwlkuser
db-password=verystrong
db-host=localhost
db-port=5432
enable-tftp=Y

To control the application directly, you should use:

/usr/sbin/spacewalk-service [stop|start|restart]

To display all the third-party services that contribute to the application:

spacewalk-service status

Option two, automatic installation


After connecting the repositories, we put up the setup, and run:

yum -y install spacewalk-setup-postgresql
yum -y install spacewalk-postgresql
spacewalk-setup

Again, questions regarding the database and SSL will be asked, using the --answer-file key and the path to the file with answers.

Base & Child Channels, Repos


To manage client stations, Spacewalk uses a system of so-called channels, which can be both primary (Base) and child (Child), each channel is assigned a repository that you need, as well as a key — through which the client is paired with server.

As a result, the repositories are synchronized with the channels, which, in turn, are connected with the clients, and Spacewalk works in a general way. It is also worth mentioning the errata, which can be tied to the channels, which simplifies the update and control packages.

All clients can be combined into groups according to different signs, and with both the same and different channels or repositories, it is possible to work with a large number of clients at times, which greatly helps to make updates on 100+ servers.

A list of all installed packages is available, and after synchronization of repositories and possible ones for installation. Some items are intuitive and there’s no point in looking at each item line by line.

All actions in Spacewalk occur on schedule (Schedule), almost any action can be customized for a time that is most convenient for you.

After installation, you can go to the address of your server, conduct further configuration through a graphical interface:



Enter the password, administrator name, organization name (which is also one of the client management methods), and go to the launch pad.

For the time being, you can either study the available opportunities on your own, or proceed to create channels.

Channels - Manage Software Channels - Create Channel:



I recommend setting the channel names according to the types and type of OS that will be associated with this channel, for example, CentOS_7_x86_64 , you can safely choose sha256 as a check, the Channell Summary field is intended for a small description of the channel. Also, you can optionally provide additional information.

Now we will create a child channel and link it to the main one. We are going in the same way as creating the main channel, only in the Parent Channel field we indicate the channel that has already been created.

We tie the repositories to the main and daughter channels.
Channels - Manage Repositories



For the main channels I use Base Resources, for the children - Updates.

After creating the repositories, you need to connect them to the channels.
Channels - Manage Software Channels, open your main channel, open the Repositories section.



There will be all the created repositories, select the one you need, check, and click on Update Resources.

Next, open the Sync subsection:



Where you can synchronize the repository and the channel, set up a schedule for synchronization.

Note: in my case of playful hands, I had to delete the standard repository from the client system.

Client installation, client pairing with the server, key management, configuration channel setup


As I mentioned earlier, Spacewalk uses the key system from Red Hat Network, through which pairing and management takes place.

To create a key, go to Systems - Activation Keys - Create Key:



Everything is very simple here, is it worth mentioning that you can set the key yourself, its format will always be 1-ХХХХХ , and also, each key will be linked to the channel. The Universal Default flag causes new systems to pick up key parameters.

Client installation We will

connect the client repository and install the necessary packages:

rpm -Uvh https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/spacewalk-2.8-client/epel-7-x86_64/00742644-spacewalk-repo/spacewalk-client-repo-2.8-11.el7.centos.noarch.rpm
yum -y install rhn-client-tools rhn-check rhn-setup rhnsd m2crypto yum-rhn-plugin rhncfg-actions deltarpm


Note: rhncfg-actions and deltarpm are needed for correct configuration and remote control.

Spacewalk uses rhn_check to synchronize a client server that runs every 4 hours. This value can be reduced to 60 minutes, but for me neither one nor the other options were optimal, therefore there are two options: use osad, which comes with Spacewalk, or simply add a cronjob for rhn_check, for example, for every minute while we we test the system, then the value can be changed as you please.

crontab -e
* * * * * /usr/sbin/rhn_check

It would also be nice to add / usr / bin / rhn-actions-control --enable-all to Kron, too, it is used for deploying configurations, and sometimes tupit.

Go back to key management, and copy the key ID that we created, and execute:

rhnreg_ks --serverUrl=http://your-server-ip/XMLRPC --activationkey=1-YOURKEY --force

Just do not forget to change the values ​​of IP and key on your own. Again, I advise you to use the --force flag , since I noticed problems without using this flag.

We return to Systems - All , satisfied notice our system. It can now be opened and explored what is there and how, but for now I strongly recommend creating a group for more convenient system management.

Systems - System Goups - Create Group , fill in the name and description, save, open the newly created group, go to the Systems section and add the system to the group.

Now let's subscribe the system to the channel, Systems - Your system - Software folder - Software Channels subfolder:



Choose your channel and click on confirmation. It is possible to try to install the package for the sake of interest, Software Packages - Install .

Configuration Channels

To manage configurations between a server client / local machine, as well as remote control, it is worth setting up a configuration channel and linking the system to it.
Go to Configuration - Configuration Channels - Create Config Channel , set the name, description, save, then in Configuration - Manage Configuration Channels - Subscribe to Channels and subscribe the configuration channel to the software channel and to the system.



Now we can deploy configurations from both the server and local machines, and create partitions.

Configuration - Add Files - Create File / Upload File:



We can also send remote commands in the form of a bash script: I



remind you that all actions go through a schedule (Schedule), you can see a list of all actions applied to this system in the Events:



Errata section

One of the most important functions in Spacewalk is errata support, which is conveniently tied to channels and allows you to monitor the level of importance of the latest updates. This is configured directly on the server, in addition to the scripts, you need to download the necessary packages for Perla:

yum -y install perl-Frontier-RPC perl-Text-Unidecode
wget https://raw.githubusercontent.com/stevemeier/cefs/master/errata-import.pl
chmod +x errata-import.pl

Then we create the main update script and the errata collection, which will be stored in / etc / rhn /:

#!/bin/bash
cd /etc/rhn/
wget -N http://cefs.steve-meier.de/errata.latest.xml
wget -N https://www.redhat.com/security/data/oval/com.redhat.rhsa-all.xml
export SPACEWALK_USER='root'
export SPACEWALK_PASS='verystrong'
./errata-import.pl --server  YourServerIPAddress --errata errata.latest.xml --rhsa-oval  com.redhat.rhsa-all.xml --publish
unset SPACEWALK_USER
unset SPACEWALK_PASS

In the export, enter the name and password of the Spacewalk administrator that you specified at the very beginning.

Make the script executable, and add it to Cron:

chmod +x spcwlk_errata.sh
crontab -e
0 2 * * 7 /usr/bin/sh /etc/rhn/spcwlk_errata.sh

Run it right now to see the changes in the GUI. The execution will take some time.



Total


An important point, personally in my opinion, which I missed, is setting up and installing a proxy, as well as raising the entire system like HA. I had an idea to configure the application via pacemaker, and to synchronize on the database. As a result, it was decided to abandon this idea, in view of the not very large server fleet of client stations. However, if the system in the next six months shows itself to be extremely useful, you may have to expand the original server.

Automation


Server:

Bash script for server deployment (DISABLES firewalld):

#!/bin/sh
#
rpm -Uvh https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/spacewalk-2.8/epel-7-x86_64/00736372-spacewalk-repo/spacewalk-repo-2.8-11.el7.centos.noarch.rpm
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
(cd /etc/yum.repos.d && curl -O https://copr.fedorainfracloud.org/coprs/g/spacewalkproject/java-packages/repo/epel-7/group_spacewalkproject-java-packages-epel-7.repo)
yum -y install spacewalk-setup-postgresql spacewalk-postgresql wget perl-Frontier-RPC perl-Text-Unidecode perl-XML-Simple
#
mkdir /usr/share/spcwlk-tmp/
echo 'admin-email = root@localhost
ssl-set-cnames = spcwlkserver
ssl-set-org = Unicorn
ssl-set-org-unit = EOH
ssl-set-city = Prague
ssl-set-state = SCK
ssl-set-country = CZ
ssl-password = verystrong
ssl-set-email = root@localhost
ssl-config-sslvhost = Y
db-backend=postgresql
db-name=spcwlkdb
db-user=spcwlkuser
db-password=verystrong
db-host=localhost
db-port=5432
enable-tftp=Y' > /usr/share/spcwlk-tmp/spcwlk_answer
spacewalk-setup --answer-file=/usr/share/spcwlk-tmp/spcwlk_answer
#
systemctl stop firewalld
systemctl disable firewalld
#


Bash script to install errata (first set the name and password for the Spacewalk administrator):
#!/bin/sh
#
wget https://raw.githubusercontent.com/stevemeier/cefs/master/errata-import.pl -P /etc/rhn/
echo '#!/bin/bash
cd /etc/rhn/
wget -N http://cefs.steve-meier.de/errata.latest.xml
wget -N https://www.redhat.com/security/data/oval/com.redhat.rhsa-all.xml
export SPACEWALK_USER='gui username'
export SPACEWALK_PASS='Password to your gui account'
./errata-import.pl --server  YourServerIPAddress --errata errata.latest.xml --rhsa-oval  com.redhat.rhsa-all.xml --publish
unset SPACEWALK_USER
unset SPACEWALK_PASS' > /etc/rhn/spcwlk_errata.sh
#
chmod +x /etc/rhn/errata-import.pl
chmod +x /etc/rhn/spcwlk_errata.sh
#
echo '#!/bin/bash
#
/etc/rhn/./spcwlk_errata.sh' > /etc/rhn/spcwlk_errata_cron.sh
chmod +x /etc/rhn/spcwlk_errata_cron.sh
echo '0 2 * * 7 /usr/bin/sh /etc/rhn/spcwlk_errata_cron.sh' >> /var/spool/cron/root
/etc/rhn/./spcwlk_errata.sh
#


Ansible playbook for server deployment (do not forget to add the answer-file):

- hosts: spcwlk-server
  tasks:
   - name: Install Spacewalk repo
     yum:
      name: https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/spacewalk-2.8/epel-7-x86_64/00736372-spacewalk-repo/spacewalk-repo-2.8-11.el7.centos.noarch.rpm
      state: present
   - name: Install epel repo
     yum:
      name: https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
      state: present
   - name: Install PostgreSQL packages
     yum:
      name:
      - spacewalk-setup-postgresql
      - spacewalk-postgresql
      - wget
      - perl-Frontier-RPC
      - perl-Text-Unidecode
      - perl-XML-Simple
   - name: Creates directory for Spacewalk answer file
     file:
       path: /usr/share/spcwlk-tmp/
       state: directory
       mode: 0755
   - name: Deploy answer file
     copy:
       src: /etc/ansible/spcwlk_answer
       dest: /usr/share/spcwlk-tmp/spcwlk_answer
   - name: Spacewalk Server Deploy
     shell: spacewalk-setup --answer-file=/usr/share/spcwlk-tmp/spcwlk_answer
   - name: Stop firewalld
     systemd:
       name: firewalld
       state: stopped
       enabled: no


Customer:

Bash script for client deployment (do not forget about IP and key):
#!/bin/sh
#
rpm -Uvh https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/spacewalk-2.8-client/epel-7-x86_64/00742644-spacewalk-repo/spacewalk-client-repo-2.8-11.el7.centos.noarch.rpm
yum -y install rhn-client-tools rhn-check rhn-setup rhnsd m2crypto yum-rhn-plugin rhncfg-actions deltarpm wget
echo '#!/bin/bash
#
/usr/sbin/rhn_check' > rhn_check.sh
mv rhn_check.sh /etc/cron.hourly/
chmod +x /etc/cron.hourly/rhn_check.sh
echo '* * * * * /usr/bin/sh /etc/cron.hourly/rhn_check.sh' >> /var/spool/cron/root
/usr/bin/rhn-actions-control --enable-all
cd /usr/share/rhn/
wget http://YourServerIPAddress/pub/RHN-ORG-TRUSTED-SSL-CERT
rhnreg_ks --serverUrl=http://172.22.64.41/XMLRPC --activationkey=1-xxxxxxxxxxxxxxxxxxxxxxxxxxxx --force


Anisble Playbook for client deployment:

- hosts: spcwlk-clients
  tasks:
    - name: Install spacewalk repo
      yum:
       name: https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/spacewalk-2.8-client/epel-7-x86_64/00742644-spacewalk-repo/spacewalk-client-repo-2.8-11.el7.centos.noarch.rpm
       state: present
    - name: Install client packages
      yum:
        name:
          - rhn-client-tools
          - rhn-check
          - rhn-setup
          - rhnsd
          - m2crypto
          - yum-rhn-plugin
          - rhncfg-actions
          - deltarpm
          - wget
    - name: Create cronjob for rhn_check
      cron:
        name: "rhn_check"
        minute: "*"
        hour: "*"
        day: "*"
        month: "*"
        weekday: "*"
        job: "/usr/sbin/rhn_check"
    - name: Enable controls for config and remote control deployment
      shell: /usr/bin/rhn-actions-control --enable-all
    - name: Get certificate from server to client
      get_url:
        url: http://YourServerIPAddress/pub/RHN-ORG-TRUSTED-SSL-CERT
        dest: /usr/share/rhn/
    - name: Register client to server
      rhn_register:
        state: present
        server_url: http://YourServerIPAddress/XMLRPC
        activationkey: "{{ activation_key }}"


Thank you all for reading the article. Good luck!