Cluster storage in Proxmox. Part three. Nuances


    The third part of the article is a kind of appendix to the two previous ones, in which I talked about working with the Proxmox cluster. In this part, I will describe the problems that we encountered in working with Proxmox, and how to solve them.

    Authorized iSCSI Connection

    If you need to specify creditsyal when connecting to iSCSI - it is better to do this bypassing Proxmox . Why?

    • Firstly, because it is not possible to create an authorized iSCSI connection through the Proxmox web interface .
    • Secondly, even if you decide to create an unauthorized connection in Proxmox in order to specify the authorization information manually, you will have to hang out with the system for the ability to change the target configuration files, because if the connection to the iSCSI host fails, Proxmox overwrites the target information and retries the connection.

    Easier to connect manually :

    root@srv01-vmx:~# iscsiadm -m discovery -t st -p
    root@srv01-vmx:~# iscsiadm -m node --targetname "iqn.2012-10.local.alawar.ala-nas-01:pve-cluster-01" --portal "" --op=update --name node.session.auth.authmethod --value=CHAP
    root@srv01-vmx:~# iscsiadm -m node --targetname "iqn.2012-10.local.alawar.ala-nas-01:pve-cluster-01" --portal "" --op=update --name node.session.auth.username --value=Admin
    root@srv01-vmx:~# iscsiadm -m node --targetname "iqn.2012-10.local.alawar.ala-nas-01:pve-cluster-01" --portal "" --op=update --name node.session.auth.password --value=Lu4Ii2Ai
    root@srv01-vmx:~# iscsiadm -m node --targetname "iqn.2012-10.local.alawar.ala-nas-01:pve-cluster-01" --portal "" --login

    These commands must be executed for all portals that provide the target we need on all nodes of the cluster. Or you can execute these commands on one node, and distribute the configuration files for the connection to the rest. Files are located in the directories " / etc / iscsi / nodes " and " / etc / iscsi / send_targets ".

    Mounting on the new GFS2-FS node

    In order to mount a GFS2- file system on a new node , it ( file system ) needs to add one more log. This is done as follows: on any node of the cluster on which the FS we need is mounted , the command is executed:

    root@pve01:~# gfs2_jadd -j 1 /mnt/cluster/storage01

    The " -j " parameter specifies the number of logs to add to FS .

    This command may fail:

    create: Disk quota exceeded

    Causes of the error:

    Inside the GFS2 volume there is actually not one file system, but two. Another file system is for business purposes. If desired, it can be mounted by adding the option " -o meta ". Changes within this FS could potentially lead to the destruction of the data file system. When adding a log to FS , the meta- file system is mounted in the " / tmp / TEMP_RANDOM_DIR " directory , after which a log file is created in it. For reasons that are not yet clear to us, the kernel sometimes believes that in mounted meta-FSThe quota for creating objects has been exceeded, which is why such an error occurs. You can get out of this situation by remounting GFS2 with data (of course, for this you need to stop all the virtual machines located on this FS ) and run the add log command again. You also need to unmount meta-FS from the last unsuccessful attempt to add a log:

    cat /proc/mounts | grep /tmp/ | grep -i gfs2 | awk '{print $2}' | xargs umount

    Mounting a data source inside a container

    Container virtualization technology is good because the host has almost unlimited possibilities for communicating with the virtual machine.

    When the container starts, vzctl tries to execute the following set of scripts ( if any ):

    • /etc/pve/openvz/vps.premount
    • /etc/pve/openvz/CTID.premount
    • /etc/pve/openvz/vps.mount
    • /etc/pve/openvz/CTID.mount
    • /etc/pve/openvz/CTID.start

    When stopped, the following scripts are executed:

    • /etc/pve/openvz/CTID.stop
    • /etc/pve/openvz/CTID.umount
    • /etc/pve/openvz/vps.umount
    • /etc/pve/openvz/CTID.postumount
    • /etc/pve/openvz/vps.postumount

    where " CTID " is the container number. The " vps. * " Scripts are executed during operations with any container. The scripts " * .start " and " * .stop " are executed in the context of the container, all the rest in the context of the host. Thus, we can script the start / stop process of the container by adding data mounting to it. Here are some examples:

    Mounting a data directory inside a container

    If the container works with a large amount of data, we try not to keep this data inside the container, but mount it from the host. There are two positive aspects to this approach:

    1. The container is small, quickly backed up by Proxmox . We have the opportunity at any time to quickly restore / clone the functionality of the container.
    2. Container data can be centrally backed up by an adult backup system with all the amenities provided by it (multilevel backups, rotation, statistics, and so on).

    The contents of the file " CTID.mount ":

    . /etc/vz/vz.conf        # подключаем файл с глобальным описанием внутренних переменных OpenVZ. В частности, в нем определена переменная ${VE_ROOT} - корневой каталог контейнера на хосте.
    . ${VE_CONFFILE}         # подключаем файл с описанием переменных контейнера
    DIR_SRC=/storage/src_dir # каталог на хосте, который надо примонтировать внутрь контейнера
    DIR_DST=/data            # каталог внутри контейнера, к которому будет примонтирован $DIR_SRC
    mkdir -p ${VE_ROOT}/${DIR_DST} # создаем внутри контейнера каталог назначения
    mount -n -t simfs ${DIR_SRC} ${VE_ROOT}/{$DIR_DST} -o /data  # монтируем каталог внутрь контейнера

    Mounting the file system inside the container

    There is one on the host that needs to be given to the container. The contents of the file " CTID.mount ":

    . /etc/vz/vz.conf
    . ${VE_CONFFILE}
    UUID_SRC=3d1d8ec1-afa6-455f-8a27-5465c454e212 # UUID тома, который надо примонтировать внутрь контейнера
    mkdir -p ${VE_ROOT}/${DIR_DST}
    mount -n -U ${UUID_SRC} ${VE_ROOT}/{$DIR_DST}

    Mounting the file system in the file inside the container

    Why would you need this? If some tricky product ( for example, Splunk ) does not want to work with simfs in any way , or we are not satisfied with the speed of GFS2 under certain conditions. For example, we have some kind of cache on a heap of small files. GFS2 does not work very quickly with large volumes of small files. Then you can create a file system on the host other than GFS2 ( ext3 ) and connect it to the container.

    Mount the loop device from the file to the container:

    First, create the file:

    root@srv01:/storage# truncate -s 10G CTID_ext3.fs

    Format FS in a file:

    root@srv01:/storage# mkfs.ext3 CTID_ext3.fs
    mke2fs 1.42 (29-Nov-2011)
    CTID_ext3.fs is not a block special device.
    Proceed anyway? (y,n) y

    The contents of the file " CTID.mount ":

    . /etc/vz/vz.conf
    . ${VE_CONFFILE}
    CFILE_SRC=/storage/CTID_ext3.fs # путь к файлу, который надо примонтировать внутрь контейнера
    mkdir -p ${VE_ROOT}/${DIR_DST}
    mount -n ${CFILE_SRC} -t ext3 ${VE_ROOT}/{$DIR_DST} -o loop

    Unmounting external data in the container when stopped

    When the container stops, the system automatically tries to unmount all file systems connected to it. But in particularly exotic configurations, she cannot do this. Therefore, just in case, an example of a simple script " CTID.umount ":

    . /etc/vz/vz.conf
    . ${VE_CONFFILE}
    if mountpoint -q "${VE_ROOT}${DIR}" ; then
            umount ${VE_ROOT}${DIR}

    Work in a cluster with a non-clustered file system

    If for some reason there is no desire to use a cluster FS (it does not suit the stability of work, it does not suit the performance, etc. ), but you want to work with a single storage, then this option is possible. For this we need:

    • Separate logical volume in CLVM for each cluster node
    • Primary storage for container operation
    • Empty backup storage for urgent mounting of a foreign node volume in case of its failure / shutdown

    Procedure: For

    each node in the cluster, we select our logical volume in CLVM , format it.

    We create the main storage. Create a directory with the same name on all nodes of the cluster ( for example, "/ storage" ). We mount our logical volume into it. In the Proxmox admin panel, create a repository of the type “ Directory ”, name it, for example, “ STORAGE ”, and say that it is not shared.

    Create backup storage. Create a directory with the same name on all nodes of the cluster ( for example, "/ storage2" ). In the Proxmox admin panel, create a repository of the type " Directory ", name it, for example, "", we say that it is not general. In the event of a fall / shutdown of one of the nodes, we will mount its volume in the" / storage2 " directory on that node of the cluster that will take the burden of the deceased.

    What do we have as a result:

    • Migration ( including online ) of containers between nodes ( if no data is mounted to the container from the side ). The container is transferred from node to node by copying, respectively, the migration time depends on the amount of data in the container. The more data, the longer the container will be transported between nodes. Do not forget about the increasing disk load at the same time.
    • ( Under- ) fault tolerance. When a node falls, its data can be mounted on a neighboring node, and theoretically, you can start working with them.

    Why " under- " and why " theoretically ":

    Virtual machines live in the " STORAGE " storage , which is located in the " / storage " directory . The disk from the dead node will be mounted in the " / storage2 " directory , where Proxmox will see the containers, but will not be able to start them from there. In order to raise the virtual machines located in this storage, you need to do three actions:

    1. To inform the fire victims that their new home is not the " / storage " directory, but the / storage2 directory . To do this, in each file " * .conf " in the directory " / etc / pve / nodes / dead_name / openvz ", change the contents of the VE_PRIVATE variable from " / storage / private / CTID " to " / storage2 / private / CTID ".
    2. Tell the cluster that the virtual machines out of that inanimate node are now located on this living one. To do this, just transfer all the files from the directory / etc / pve / nodes / dead_name / openvz to the directory / etc / pve / nodes / live_node / openvz . Perhaps there is some kind of correct API instruction for this , but we didn’t bother with this :)
    3. Reset the quota for each burnt container ( just in case ):

      vzquota drop CTID

    All. You can run containers.

    If the containers from the dead node take up little space, or we have incredibly fast disks, or we can afford to wait, then you can avoid the first and third steps by simply moving the containers we need from " / storage2 / private " to " / storage / private "

    If the cluster fell apart

    A cluster is a moody creature, and there are times when it comes to a pose. For example, after massive problems with the network, or due to a massive power outage. The pose looks like this: when accessing the cluster storage, the current session is blocked, polling the status of the fence domain displays alarm messages of the form " wait state messages ", and connection errors are poured into dmesg .

    If no attempts to revive the cluster lead to success, then the simplest thing is to disable automatic entry to the fence domain on all nodes of the cluster ( file "/ etc / default / redhat-cluster-pve"), and sequentially reload all the nodes. You must be prepared for the fact that the nodes can not reboot on their own. When all the nodes are rebooted, manually connect to the fence domain, start CLVM , and so on. Previous articles have written how to do this.

    That’s probably all.

    In the next part I will talk about how we automate the work in the cluster.

    Thanks for attention!

    Also popular now: