Introducing the xCAT Deployment and Maintenance System

Continuing the theme of our previous article , today we’ll talk about the tool that we use every day.

If your activity is somehow related to the installation and configuration of large batches of equipment, then the material, in our opinion, will be useful and interesting to you.

Any engineer who manages the Badoo server fleet , the most popular dating network on the Internet, doesn’t even want to remember that once you had to manually edit the DHCP server configurations, collect images for the PXE network download ... We want to tell you where and how Our company successfully uses the xCAT software solution.

First of all, we denote the initial conditions under which we will describe the situation:

• the main distribution - SLES 11 or SLES 11 SP 1 (mainly the second);
• the latest stable version of xCAT , a ready-made package for the specified distribution (the installation process is described in detail on the product home page);
• multiple VLANs ;
• a large number of new or existing servers.

Returning to our first article, which we also recommend reading , we will stop at the place where we need to prepare the configuration of the DHCP server, or rather, add information about new servers to it. From this moment we begin to actively use xCAT.

xCAT is a server fleet management system that has the following features:

• support for the current status of the DHCP server,
• maintaining the current state of DNS zones and performing dynamic updates,
• support for the current status of the TFTP server (which is very important for us),
• creating images for installing the OS,
• PXE server support ,
• support of the current state of static routes on servers,
• displaying the current status of any previously configured server,
• management of server groups based on the logic we specified.

The biggest advantage of the xCAT system is that it is not a completely new product, but in some way a synthesis of all previously known products, i.e. during operation, the engineer does not have to learn a new syntax, which undoubtedly affects the speed of mastering the program.

Briefly describe the process of initial product setup .

1. We go to the project website, select the appropriate distribution, get a description of the installation process.
2. Install the required packages.
3. Install the xCAT server directly.

Then proceed to set up .

The main commands for managing (adding, modifying, checking input): tabedit and tabdump Edit the

main system configuration file:
tabedit site

Key points to pay attention to:

"installdir","/install"- the directory in which xCAT will store all the system images for installation, there are various post scripts. In case of http-installation of the system, the directory will be accessible through the web interface. It will also contain templates for automatic installation of the system.
"ipmiretries","3"- Attempts to connect to the server management interface.
"ipmitimeout","2"- No comments.
"master","master_server_name"- the name of the master server that should be resolved should also have it in the file / etc / hosts.
"tftpdir","/tftpboot"- the path to the directory where the files necessary for downloading via PXE will be located, also the TFTP server must have access to the directory, because it will be he who will distribute the images for pxe-boot.
"xcatconfdir","/etc/xcat"- path to the directory with xCAT configuration files.
"timezone","GMT"- time zone.
"useNmapfromMN","yes"- use nmap on the master server to display the status of hosts (including nodestat hostname)
"dhcpinterfaces","ethN,!remote!"- the interfaces on which the DHCP server responds to DHCP requests. (The first is the physical interface, the second is for VLAN, where our server is registered as the issuing address)
"nameservers","1.1.1.1"- the addresses of DNS servers, where after issuing lease our server will try to send information about the PTR record for the host.

Such a configuration is quite sufficient for familiarization, as well as for a test run of the xCAT server.

Now we are trying to add access to an unprivileged user.

For this it is necessary:
- the presence of the user in the system;
- issue and sign certificates to the xCAT server.
/opt/xcat/share/xcat/scripts/setup-local-client.sh username

We get the $ HOME / .xcat directory , with the following contents: - add user information to the xCAT service tables. In this case, we give the user maximum permissions, for this we execute and add a line of the following form: After the actions we have done, the user gains access to the xCAT server management. Next, to create the installation repository, we need to have images of operating systems
ca.pem
client-cert.pem
client-cred.pem
client-key.pem
client-req.pem


tabedit policy


username,allow



which we are going to use.

XCAT includes copycds utility, which works according to the following scenario:
copycds [{-n|--name|--osver}=distroname] [{-a|--arch}=architecture] 1st.iso [2nd.iso …]

Thus, if we have an image of the installation DVD, then simply executing
copycds PATH_TO_ISO SLES-11-DVD-x86_64-GM-DVD1.iso

xCAT will independently detect the version of the operating system (if not --osver ), then copy everything to the local disk ( in our case - / install / sles11 )

xCat also has a set of configurations for standard installation, which are located in the / opt / xcat / share / xcat / install / directory , but are of no interest to us, so we create our installation profile and place it along the path
/ install / custom / install / DISTRNAME / Tem platename.tmpl .

Note: in the case of SuSe Linux, this is an xml file that YaST will help us to do (although it is easier to write it yourself, after having familiarized yourself with the structure and composition of the parameters on the manufacturer’s website).

Once the typical installation template is ready, it remains only to add new machines and start installing the OS , which will be discussed below.

For the correct installation procedure, we need information:
1) about the servers on which we are going to install the OS,
2) the hostname of each server,
3) about the subnets that we plan to use,
4) about the correspondence of hostame-ip,
5) about access details to access server management interface.

Returning to our previous article, we recall that we have all the necessary information, we just need to add it to xCAT, which we demonstrate:

1. Add server groups to the system:
nodeadd depl[1-200] groups=depl

2. Add information about server management interfaces (suppose they are named like this: depl [ 1-200] ipmi ):
nodeadd depl[1-200]ipmi groups=depl_ipmi

3. We talked about the fact that we have a subnet (VLAN) in which all the new equipment is located. We describe the subnets as follows: Note: since xCat configures the DHCP server, and stores information about fixed addresses in a file with leases , it is advisable to use fixed addresses of hosts that do not fall into the dynamic range, in order to avoid possible problems with the loss of servers in subnets.

#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,logservers,dynamicrange,nodehostname,comments,disable

tabedit networks

"depl_vlan","2.2.2.0","255.255.255.0","!remote!","2.2.2.1",,"1.1.1.1","5.5.5.5","6.6.6.6",,"2.2.2.200-2.2.2.254",,,
"depl_vlan_ipmi","3.3.3.0","255.255.255.0","!remote!","3.3.3.1",,"1.1.1.1","5.5.5.5","6.6.6.6",,"3.3.3.200-3.3.3.3.254",,,



4. Add information about the MAC addresses of the main server interfaces, as well as the management interfaces: Hint: in the last article we talked about the nodes file , with which we work closely, we will add information about MAC addresses to it, then it will be enough to execute and get the output of the form of the same for the management interfaces: 5. Add hostame - ip correspondence information: We can generate these settings using a simple command: 6. We correct the options for virtual consoles and their speeds (here we will need the accessory of all new ma tires to one group: in the end, one row will be enough for us instead of two hundred.): 7. Edit the noderes table :

#node,interface,mac,comments,disable

tabedit mac

"depl1","eth0","MAC1",,
"depl2","eth0","MAC2",,
"depl3","eth0","MAC3",,
"depl1ipmi","eth0","ipmi_MAC1",,
"depl2ipmi","eth0","ipmi_MAC2”,,
"depl3ipmi","eth0","ipmi_MAC3",,
...



cat nodes | awk {'print "\""$1"\",\"eth0\",\""$15"\""'}



"depl1","eth0","MAC1",,



cat nodes | awk {'print "\""$1"manage\",\"eth0\",\""$10"\""'}

"depl1ipmi","eth0","ipmi_MAC1",,



#node,ip,hostnames,otherinterfaces,comments,disable

tabedit hosts

"depl1","2.2.2.2","depl1","depl1ipmi:3.3.3.2",,
"depl2","2.2.2.3","depl2","depl2ipmi:3.3.3.3",,
"depl3","2.2.2.4","depl3","depl3ipmi:3.3.3.4",,
...


for i in `seq 1 200`; do echo '"depl$i'","2.2.2.'$[$i+1]'","depl'$i'","depl'$i'manage:3.3.3.'$[$i+1]'",,'; done



#node,power,mgt,cons,termserver,termport,conserver,serialport,serialspeed,serialflow,getmac,comments,disable

tabedit nodehm

"testgroup","ipmi","ipmi",,,,"1","1","115200",,,,



#node,servicenode,netboot,tftpserver,nfsserver,monserver,nfsdir,installnic,primarynic,discoverynics,cmdinterface,xcatmaster,current_osimage,next_osimage,nimserver,comments,disable

tabedit noderes

"depl",,"pxe","1.1.1.1","1.1.1.1",,,,,,,,,,,,

8. Add information about the type of our servers, as well as the OS installation method: In this case, we use the install method - system installation, you can use netboot - for network boot, as well as for performing service procedures. 9. Add information about the correspondence of the main server interface and its management interface: Here we also cheat and will not indicate separately the correspondence for each server, but use the group. Given that the naming of the management interface is nothing more than adding ipmi to the hostname of the server, this is exactly what we need. And we take into account that we have the same username & password for IPMI for all servers, which is also indicated in the table. 10. Now we generate a file configuration

#node,os,arch,profile,provmethod,supportedarchs,nodetype,comments,disable

tabedit nodetype

"depl","sles11.1","x86_64","Templatename","install",,"osi",,





#node,bmc,bmcport,username,password,comments,disable

tabedit ipmi

"depl","/\z/ipmi/","0",”ipmi_username”,"ipmi_password",,



/ etc / hosts :
makehosts

11. Create a configuration for the DHCP server:
makedhcp -a -n

You can view information on any host:
lsdef nodename

Now the server is ready to start the installation process.

12. We carry out :
nodeset depl[1-10] install

Instead of depl [1-10] it is allowed to use the name of the group or a comma separated list.
At this point, we get the configurations for PXE boot to each host in /tftpboot/pxelinux.cfg/ and a copy of the autoinstall file in / install / autoinst / .

13. We give the server a command to boot from the network interface at the next boot
rsetboot depl[1-10] net

14. We send the server to reboot
rpower depl[1-10] boot

Next, we can observe the installation process in the server consoles, or configure conserver(This feature is not considered under this article).

Last of all, we collect the image for network boot . In Badoo, images are used to diagnose, restore, and revitalize problem equipment, prepare custom hardware, and perform performance tests.

To do this, we run the genimage command - it will launch a simple wizard, answering the questions of which we will get the system image in.
/install/netboot/$DISTR/$ARCH/profile_name

The resulting image can be changed and edited at our discretion, after which we should execute.
packimage -o $DISTR -p profile_name -a $ARCH

So we get a ready netboot image to which we send our servers: To netboot image it is advisable to put public keys, but you can use the details for access from the passwd table (

nodeset depl[11-200] netboot=$DISTR-$ARCH-profile_name

rsetboot depl[11-200] net

rpower depl[11-200] boot

tabedit passwd )

After all the preparatory procedures are completed on the servers, we start installing the OS on them - we will get the result in 15 minutes.

It remains only to place all our servers in the necessary subnets, after which they go through Puppet , which will install the necessary software individually, in groups, and also set the necessary system parameters.

In conclusion, I would like to note that if there is a system of this kind, it is enough for the engineer to give the necessary commands, making a minimum of effort to configure the servers.

Of course, today we did not talk about all the advantages of the xCAT system and the intricacies of the settings, such issues as managing hypervisors, as well as virtual servers on them through xCAT, remained outside the scope of the article; xCAT server installation and management via the web interface, access to server consoles from the management server.

We will be glad to continue the series of articles on the above topics if they cause your interest, which you can always tell us in the comments.