January 24, 2013 at 17:52Cluster storage in Proxmox. Part two. Launch
Hello! 
This is the second part of the article about working with cluster storage in Proxmox . Today we’ll talk about connecting storage to the cluster.
In the beginning I want to give an excerpt from a previous article so that no one will forget why we need the whole garden with a cluster:
As a cluster file system, we use Global File System 2 .
GFS2 in Proxmox is functional, but Proxmox developers do not officially support it. However, the core of Proxmox is based on the RedHat core of the RHEL6x branch. That is, support for GFS2 in the kernel is very mature. The binding also behaves quite stably, with the exception of a few nuances, which I will discuss later. The gfs2-utils package itself is practically unchanged ( there are only patches in the starting scripts for fitting for the Debian specifics) stable redhat version of gfs2-utils-3.1.3
Packagegfs2-utils appeared in Proxmox in February 2012 . The native Debian gfs2-tools package wildly conflicts ( which is not surprising ) with the entire RedHat cluster from Proxmox , therefore, before Proxmox version 2.0, it was completely out of the box from GFS2 . So, a huge plus is that the foundation for cocking GFS2 in Proxmox is already built. As an iSCSI storage we use HP MSA 2012i
. This machine is a fault-tolerant solution based on the use of an array of hard drives connected to two independent raid- controllers. Each raid- controller has two interfaces for data transfer, in the framework of this article it is interesting in that the controller does not know how to combine these interfaces . To load both controller interfaces, we will use multipath . I will not describe the creation of volumes. Volumes are created without any authorization ( I will discuss the features of an authorized connection from Proxmox to iSCSI in the next article ).
It is advisable to configure jumbo frames .
To work with several network storage interfaces, we configure multipath . We create the file /etc/multipath.conf with the following contents:
The blacklist block devices that are to be excluded from processing fall ( local drive ). In our case, these are cciss devices, which are the volumes of the HP Smart Array controller served by the cciss kernel module .
The " user_friendly_names " parameter allows you to create user-friendly devices in / dev / mapper like " mpath0-part1 ".
Install the missing packages:
The installed multipath immediately takes off and joyfully picks up the config.
Prepare the open-iscsi daemon. We need to automatically connect the available targets at system startup. Edit the file /etc/iscsi/iscsid.conf . We change the line in it:
on the
We configure LVM . Switch the locking method from file to cluster:
Allow the start of the CLVM . File / etc / default / clvm :
We start CLVM. If we do not have fenced configured (see the previous article ), we get the error:
CLVM does not work if our node does not belong to the fence domain.
We connect storage to a cluster.
In the admin panel we say " Add iSCSI-target ". After that, all the nodes of the cluster should see several ( in our case, two ) block devices, and multipath should make one of them, and put it in the / dev / mapper directory .
We make sure that the multipath device / dev / mapper / mpath0 is the iSCSI volume we need .
On one of the machines we mark up the storage:
In the above example, the volume is divided into two sections: one section with a capacity of 512G , and the second, taking up the remaining space on the volume.
Tom kvm01 we will need in the future, when get to configure storage for the KVM .
Restart the multipath daemon:
On the same machine, we create two clustered volume groups:
The -c option indicates that the volume group is clustered.
In principle, it was possible to create only one volume group, and keep in it both partitions for KVM machines and a GFS2 partition. This is a matter of taste.
In the CLUSTER01 group , create the Logical Volume :
On all nodes of the cluster, this Logical Volume should be visible:
We tell CLVM which Volume Groups to activate / deactivate at start / stop:
File / etc / default / clvm :
Everything is ready to create a clustered file system. We look at the name of our cluster:
The cluster name must be specified when creating the FS .
On one of the cluster nodes we format FS :
Here:
We look at the UUID of our FS :
On each node, create an entry in fstab to mount FS :
Create the directory / mnt / cluster / storage01 , mount FS in it :
There is one point. When the system is turned off, the script /etc/init.d/umountiscsi.sh is called in Proxmox when the open-iscsi daemon is stopped . It deals with disabling iSCSI- mounted file systems. To search for such systems, it uses rather complicated logic, which sometimes fails, because of which there is an attempt to unmount more than necessary, or vice versa - the necessary is not unmounted. For example, we have encountered attempts to unmount a root file system. Of course, he didn’t succeed in doing this, after which the OS entered a state of permanent wait: without stopping iSCSI targets, the system cannot reboot, and umountiscsicannot unmount all iSCSI-FS due to the fact that the root is assigned to their list.
We did not delve deeply into the logic of umountiscsi.sh . It was decided that we should not rely on umountiscsi.sh , we will manage the mounted file systems on iSCSI volumes ourselves, and the role of umountiscsi.sh will boil down to the grave report that " All systems are unmounted, my general! ".
So, in /etc/init.d/umountiscsi.sh we change the " stop " section .
It was:
It became:
Now the system will fold correctly. However, under one condition - at the time of stopping, the system should not have iSCSI- mounted file systems. If you do not want to disable FS manually, then, for example, you can unmount it in /etc/init.d/clvm before calling " stop ". At this point, all virtual machines are ( should be ) already extinguished. We ourselves do not hope for this, and before restarting, we unmount FS manually.
All that remains for us is to create a shared storage of the " Directory " type in the Proxmox admin panel , show it the path to the directory with the FS mounted , and select the " publicly available " checkbox". All OpenVZ containers created on this storage can easily migrate between nodes.
After several months of testing, we caught the kernel panic in the gfs2 module a couple of times . Fencing works fine, so at first we didn’t even understand what was happening, just the nodes rebooted several times. After switching to the new kernel version ( 2.6.32-17-pve ), there were no freezes. The new core is based on the 2.6.32-279.14.1.el6 core from RHEL6. There are some corrections regarding gfs2.
Everything is much simpler here. We have already created a volume group; it remains to set Proxmox on it . In the admin panel, create a storage of the type " LVM Group ", in the field " main storage " specify " existing partition groups ", in the field " partition group " select KVM01 , and select the " public " checkbox . For KVM machines in this section, the system will automatically create logical volumes.
Perhaps it’s worth a round off. In the next part I will talk about how you can try to use OpenVZ to live on a network storage without cluster FS, about some of the nuances in working with network attached storage, plus some solutions for automating and optimizing life in OpenVZ .
Thanks for attention!
This is the second part of the article about working with cluster storage in Proxmox . Today we’ll talk about connecting storage to the cluster.
In the beginning I want to give an excerpt from a previous article so that no one will forget why we need the whole garden with a cluster:
In our case, the problem of organizing a shared repository comes down to two aspects:
- We have a block device distributed over the network, to which several hosts will have access simultaneously. In order for these hosts not to fight for space on the device, we need the CLVM - Clustered Logical Volume Manager . This is the same as LVM , only Clustered . Thanks to CLVM, each host has up-to-date information ( and can safely change it without risking integrity ) about the state of LVM volumes on Shared Storage . Logical volumes in CLVM live exactly the same as in regular LVM . Logical volumes contain either KVM images or clusterFS .
- In the case of OpenVZ , we have the logical volume on which the file system is located. The simultaneous operation of several machines with a non-cluster file system leads to inevitable errors in the operation of everything - this is a swan, cancer and pike, only worse. The file system must be aware that it lives on a shared resource, and be able to work in this mode.
As a cluster file system, we use Global File System 2 .
GFS2 in Proxmox is functional, but Proxmox developers do not officially support it. However, the core of Proxmox is based on the RedHat core of the RHEL6x branch. That is, support for GFS2 in the kernel is very mature. The binding also behaves quite stably, with the exception of a few nuances, which I will discuss later. The gfs2-utils package itself is practically unchanged ( there are only patches in the starting scripts for fitting for the Debian specifics) stable redhat version of gfs2-utils-3.1.3
Packagegfs2-utils appeared in Proxmox in February 2012 . The native Debian gfs2-tools package wildly conflicts ( which is not surprising ) with the entire RedHat cluster from Proxmox , therefore, before Proxmox version 2.0, it was completely out of the box from GFS2 . So, a huge plus is that the foundation for cocking GFS2 in Proxmox is already built. As an iSCSI storage we use HP MSA 2012i
. This machine is a fault-tolerant solution based on the use of an array of hard drives connected to two independent raid- controllers. Each raid- controller has two interfaces for data transfer, in the framework of this article it is interesting in that the controller does not know how to combine these interfaces . To load both controller interfaces, we will use multipath . I will not describe the creation of volumes. Volumes are created without any authorization ( I will discuss the features of an authorized connection from Proxmox to iSCSI in the next article ).
Procedure
The following actions are performed on each node of the cluster.
It is advisable to configure jumbo frames .
To work with several network storage interfaces, we configure multipath . We create the file /etc/multipath.conf with the following contents:
blacklist {
devnode "cciss"
}
defaults {
user_friendly_names yes
}
The blacklist block devices that are to be excluded from processing fall ( local drive ). In our case, these are cciss devices, which are the volumes of the HP Smart Array controller served by the cciss kernel module .
The " user_friendly_names " parameter allows you to create user-friendly devices in / dev / mapper like " mpath0-part1 ".
Install the missing packages:
root@pve03:~# apt-get install multipath-tools gfs2-utils open-iscsi parted
The installed multipath immediately takes off and joyfully picks up the config.
Prepare the open-iscsi daemon. We need to automatically connect the available targets at system startup. Edit the file /etc/iscsi/iscsid.conf . We change the line in it:
node.startup = manual
on the
node.startup = automatic
We configure LVM . Switch the locking method from file to cluster:
root@pve03:~# lvmconf --enable-cluster
Allow the start of the CLVM . File / etc / default / clvm :
START_CLVM=yes
We start CLVM. If we do not have fenced configured (see the previous article ), we get the error:
root@pve03:~# service clvm start
Starting Cluster LVM Daemon: clvmclvmd could not connect to cluster manager
Consult syslog for more information
failed!
CLVM does not work if our node does not belong to the fence domain.
We connect storage to a cluster.
In the admin panel we say " Add iSCSI-target ". After that, all the nodes of the cluster should see several ( in our case, two ) block devices, and multipath should make one of them, and put it in the / dev / mapper directory .
We make sure that the multipath device / dev / mapper / mpath0 is the iSCSI volume we need .
On one of the machines we mark up the storage:
root@pve03:~# parted /dev/mapper/mpath0 mklabel gpt
root@pve03:~# parted /dev/mapper/mpath0 mkpart cluster01 0% 512G
root@pve03:~# parted /dev/mapper/mpath0 mkpart kvm01 512G 100%
In the above example, the volume is divided into two sections: one section with a capacity of 512G , and the second, taking up the remaining space on the volume.
Tom kvm01 we will need in the future, when get to configure storage for the KVM .
Restart the multipath daemon:
root@pve03:~# service multipath-tools restart
On the same machine, we create two clustered volume groups:
root@pve03:~# vgcreate -c y CLUSTER01 /dev/mapper/mpath0-part1
root@pve03:~# vgcreate -c y KVM01 /dev/mapper/mpath0-part2
The -c option indicates that the volume group is clustered.
In principle, it was possible to create only one volume group, and keep in it both partitions for KVM machines and a GFS2 partition. This is a matter of taste.
In the CLUSTER01 group , create the Logical Volume :
root@pve03:~# lvcreate -n STORAGE -l +100%Free CLUSTER01
On all nodes of the cluster, this Logical Volume should be visible:
root@srv-01:~# lvscan
ACTIVE '/dev/CLUSTER01/STORAGE' [976.56 GiB] inherit
ACTIVE '/dev/pve/swap' [4.00 GiB] inherit
ACTIVE '/dev/pve/root' [16.75 GiB] inherit
ACTIVE '/dev/pve/data' [38.21 GiB] inherit
We tell CLVM which Volume Groups to activate / deactivate at start / stop:
File / etc / default / clvm :
LVM_VGS="CLUSTER01 KVM01"
Everything is ready to create a clustered file system. We look at the name of our cluster:
root@srv-01:~# pvecm status | grep "Cluster Name"
Cluster Name: alapve
root@srv-01:~#
The cluster name must be specified when creating the FS .
On one of the cluster nodes we format FS :
root@pve03:~# mkfs.gfs2 -t alapve:storage01 -j 3 /dev/mapper/CLUSTER01-STORAGE
Here:
- "-t alapve: storage01" is the name of the lock table.
- alapve - cluster name,
- storage01 is the unique name of the file system.
- "-j 3" is the number of logs that must be created when creating FS . Usually equal to the number of nodes in the cluster. Each host that mounts FS needs its own log.
We look at the UUID of our FS :
root@srv-01:~# blkid /dev/CLUSTER01/STORAGE
/dev/CLUSTER01/STORAGE: LABEL="alapve:storage01" UUID="8b3f1110-8a30-3f2d-6486-a3728baae57d" TYPE="gfs2"
On each node, create an entry in fstab to mount FS :
root@srv-01:~# echo "UUID=8b3f1110-8a30-3f2d-6486-a3728baae57d /mnt/cluster/storage01 gfs2 noatime,_netdev 0 0" >> /etc/fstab
Create the directory / mnt / cluster / storage01 , mount FS in it :
root@srv-01:~# mount /mnt/cluster/storage01
There is one point. When the system is turned off, the script /etc/init.d/umountiscsi.sh is called in Proxmox when the open-iscsi daemon is stopped . It deals with disabling iSCSI- mounted file systems. To search for such systems, it uses rather complicated logic, which sometimes fails, because of which there is an attempt to unmount more than necessary, or vice versa - the necessary is not unmounted. For example, we have encountered attempts to unmount a root file system. Of course, he didn’t succeed in doing this, after which the OS entered a state of permanent wait: without stopping iSCSI targets, the system cannot reboot, and umountiscsicannot unmount all iSCSI-FS due to the fact that the root is assigned to their list.
We did not delve deeply into the logic of umountiscsi.sh . It was decided that we should not rely on umountiscsi.sh , we will manage the mounted file systems on iSCSI volumes ourselves, and the role of umountiscsi.sh will boil down to the grave report that " All systems are unmounted, my general! ".
So, in /etc/init.d/umountiscsi.sh we change the " stop " section .
It was:
stop|"")
do_stop
;;
It became:
stop|"")
#do_stop
exit 0
;;
Now the system will fold correctly. However, under one condition - at the time of stopping, the system should not have iSCSI- mounted file systems. If you do not want to disable FS manually, then, for example, you can unmount it in /etc/init.d/clvm before calling " stop ". At this point, all virtual machines are ( should be ) already extinguished. We ourselves do not hope for this, and before restarting, we unmount FS manually.
All that remains for us is to create a shared storage of the " Directory " type in the Proxmox admin panel , show it the path to the directory with the FS mounted , and select the " publicly available " checkbox". All OpenVZ containers created on this storage can easily migrate between nodes.
About problems
After several months of testing, we caught the kernel panic in the gfs2 module a couple of times . Fencing works fine, so at first we didn’t even understand what was happening, just the nodes rebooted several times. After switching to the new kernel version ( 2.6.32-17-pve ), there were no freezes. The new core is based on the 2.6.32-279.14.1.el6 core from RHEL6. There are some corrections regarding gfs2.
Let's move on to KVM
Everything is much simpler here. We have already created a volume group; it remains to set Proxmox on it . In the admin panel, create a storage of the type " LVM Group ", in the field " main storage " specify " existing partition groups ", in the field " partition group " select KVM01 , and select the " public " checkbox . For KVM machines in this section, the system will automatically create logical volumes.
Perhaps it’s worth a round off. In the next part I will talk about how you can try to use OpenVZ to live on a network storage without cluster FS, about some of the nuances in working with network attached storage, plus some solutions for automating and optimizing life in OpenVZ .
Thanks for attention!
- Cluster storage in Proxmox. Part one. Fencing
- Cluster storage in Proxmox. Part two. Launch
- Cluster storage in Proxmox. Part three. Nuances
