Virtuozzo Storage: Real Operating Experience, Optimization and Problem Solving Tips

This article is devoted to the actual experience of operating clusters based on Virtuozzo Storage.
For a year of active implementation and use of the platform on our hosting servers, as well as when creating clusters for our customers, we gathered a lot of tips, comments and recommendations. If you are thinking about introducing this platform, you will be able to take our experience into account when designing your cluster.

---

To be or not to be? Pros and Cons of Virtuozzo

In short, to be. Here is a list of the pros and cons we found when using Virtuozzo: the pros :

• A cluster is convenient. You can turn off the virtual machine on one server and immediately turn it on on another, no copying of data or transfer time. If the server crashes, you can immediately lift all virtual machines from the crashed server to one of the free ones.
• There is no longer the concept of free space on a particular server. All servers can use shared space, limited only by the number of disks in the cluster and your license.
• Technical support exists and responds very quickly by phone and email.
• There is a direct and only dealer in the Russian Federation, so you can purchase licenses for a legal entity in rubles.
• Virtual machines based on vz work quite efficiently and even much more efficiently than, for example, on Proxmox
• There is a convenient monitoring tool that allows you to get all the necessary information about the operation of the cluster clearly and informatively. You can freely parse data for Zabbix, Monin or Nagios.
• Thanks to readykernel, most 0-day vulnerabilities in the kernel are fixed day after day without hypervisor reboots.
• Pfcache technology saves RAM on the hypervisor (more on that below).
• Anyway, Virtuozzo Storage is a network file system, and its operation is highly dependent on network performance. But in Virtuozzo, you can use the local ssd data cache for fast reading and the local ssd log for writing. Plus, Virtuozzo Storage is trying to transfer the data associated with a particular virtual machine to the hypervisor where this virtual machine is running.
• You can have several types of disks in a cluster (ssd, hdd + ssd cache and hdd), while you can freely move virtual machines between them. In the event that the fast disks run out, your machines will automatically start using other types of disks until a place for fast ones appears.
• It is strictly contraindicated to keep disks in a local raid. The main plus for us is that copies of the data are not stored on one server, but on several at once, which is much more reliable than Raid.

Unfortunately, there are also disadvantages (:

• It is quite expensive (compared to ceph + kvm), especially on large projects and data volumes.
• Once or twice a month, one of the hypervisors can freeze for no apparent reason.
• To physically free up space in the cluster from deleted data inside virtual machines, you have to run the very difficult pcompact procedure (you can read more about this below).
• Support is trying very hard, but it cannot solve really complex issues. Typically, such problems are resolved in the next update. In the meantime, you will be prompted to upgrade and reboot (in the best tradition of The IT Crowd).
• There is live migration, but if half-open sockets are used in the container, it falls off with an error (i.e. it does not exist).
• There is automatic migration and start of virtual machines from a fallen hypervisor, but the minimum number of servers in the cluster should be 5, the replication level is 3: 2 (i.e. 3 copies of the same data block in the cluster). Automation does not save if the virtual machine has a running status, but freezes.
• The components of Virtuozzo Storage are completely unable to work normally in conditions of a small amount of completely free RAM. The banal Linux disk cache will lead to the fall of specific CSs (Virtuozzo Storage daemons) and even to the fall of virtual machines or the entire hypervisor.
• The Virtuozzo Automator control panel is not suitable for real use, rather for viewing statistics on resources and load, and other alternative control panels are not found.
• Api for automation of standard operations was not found. I had to write my own, but not everything is going smoothly with him either. In fact, we perform typical operations through the bash console, but as a result, some operations may not be performed for no apparent reason. For example, we automated the process of migrating a virtual machine from a hypervisor to a hypervisor through a series of simple steps: shutting down the container on the old hypervisor, migrating, and actually launching it on the final hypervisor. Sometimes the startup procedure does not work, as it can happen too slowly, and our self-written api just falls off. Plus, it’s not clear what to do if several tasks are launched simultaneously on the same hypervisor.

So, with the pros and cons more or less figured out, now the real recommendations :

Installation

If you want to have at least some ability to control the installation parameters, or you need to add a new node to an old cluster created six months ago (for example, when booting from a disk), select the second item - install cli.

In this case, however, you will lose the opportunity to get a beautiful cluster control panel.

If you want to add disks to the cluster at the installation stage (but do not do it better), then make sure that they are fully formatted, otherwise the installer will draw you a beautiful error in the middle of the installation.

Swap and pfcache systems require fast ssd disks. Do not miss this moment, otherwise it will be difficult to redo it. If with swap everything is more or less less clear, then what pfcache is and what it is with is not immediately clear. In fact, in all virtual machines folders are scanned along a specific path, all libraries and executable files are cached, and the hash log is located in a special local virtual disk, which, in turn, is stored on the system partition. Then, when starting any binary file inside the virtual machine, the hash log (which is on the disk) is analyzed, and if such a binary is already running, the new one does not start, but simply creates a link in memory to the existing one. Now imagine what will happen if the log disk is not ssd. And if swap will still be stored there? )

Here's how to transfer the log:

service pfcached stop
ploop umount /vz/pfcache.hdd/DiskDescriptor.xml
mv /vz/pfcache.hdd /mnt/ssd2/
sed '/^PFCACHE_IMAGE=/ s~.*~PFCACHE_IMAGE="/mnt/ssd2/pfcache.hdd"~' -i /etc/vz/pfcache.conf
ploop mount -m /vz/pfcache /mnt/ssd2/pfcache.hdd/DiskDescriptor.xml
service pfcached restart

The Internet is full of information on how to mount a file swap on a fast disk and disable the old slow swap, we will not duplicate. But be sure to consider all the recommendations on drives and networks, which we will analyze below. And also plan that there should be at least three cluster servers.

Be sure to enable automatic time synchronization during the installation phase , if you didn’t, then it’s not difficult to fix it:

yum install ntp -y
systemctl stop ntpd
ntpdate  165.193.126.229 0.ru.pool.ntp.org 1.ru.pool.ntp.org 2.ru.pool.ntp.org 3.ru.pool.ntp.org
systemctl start ntpd
systemctl enable ntpd

After installation, we immediately make a number of changes to the standard sysctl kernel settings , but I do not recommend doing this for no apparent reason. Better consult with support.

echo "fs.file-max=99999999" >> /etc/sysctl.d/99-files.conf
echo "kernel.sysrq=1" >> /etc/sysctl.d/99-reboot.conf
echo "kernel.panic=1" >> /etc/sysctl.d/99-reboot.conf
sysctl -w fs.file-max=99999999
sysctl -w kernel.panic=1
sysctl -w kernel.sysrq=1

File limit settings, bash prompts, options for screen

cat > /etc/security/limits.d/nofile.conf << EOL
root      soft    nofile           1048576
root      hard    nofile           1048576
*         soft    nofile           1048576
*         hard    nofile           1048576
*         hard    core           0
EOL
cat > /etc/profile.d/bash.sh << EOL
PS1='\[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]\$ \[\033[00m\] '
EOL
sed -i 's/Defaults\    requiretty/#Defaults\    requiretty/g' /etc/sudoers

I highly recommend having a local data cache for virtual machines , it should be on the ssd-drive. This will allow your virtual machines not to receive data through the network, but to take them locally from the cache, which will speed up the operation of the cluster and virtual machines many times over.

To do this, edit / etc / fstab, where livelinux is the cluster name, / vz / client_cache, cachesize = 50000 0 0 is the path to the ssd log with a size in MB.

vstorage://livelinux /vstorage/livelinux fuse.vstorage _netdev,cache=/vz/client_cache,cachesize=50000 0 0

Network

You need a fast network. You should not raise a cluster with a 1 Gb network, because this is a shot at once in your own leg. When there are a lot of machines and data, you will begin to have serious problems. The file system is network, the disk works through the network, balancing and restoring replication, too. If you fill the net 100%, avalanche drops and brakes will begin. In no case do you need to run a cluster on a 1 GB network, unless of course it is a test cluster.

Not a good, but possible alternative to 10 GB of optics is to make a bonding of 2 gigabit network cards. This can give a peak of up to 1.5 GB, and in very good and clear weather even 2 GB.
In this type of bonding is preferably 802.3ad. It, in turn, requires configuration on the side of your network equipment. Also, make sure you select xmit_hash_policy = layer3 + 4, as this is the most productive option (according to Virtuozzo).

Example from / etc / sysconfig / network-scripts / ifcfg-bond0

BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=802.3ad xmit_hash_policy=layer3+4"

Please note that bonding will work only if the server is working with two other servers at the same time. Those. direct exchange between only two servers will occur at a speed of 1 GB, but when there are already three servers, you can exchange data already at a speed of about 1.5 GB.

When you install the first node of the cluster, the network should already work, and Virtuozzo should be able to send broadcast requests. In other words, if you do not have an uplink in the network interface, Virtuozzo will silently stop and hang for no reason at the final installation stage. In fact, she just tries to send a request to the network via broadcast, and if she doesn’t succeed, then she stops the installation.

Well, of course, it is highly desirable for you to have separate network interfaces and networks for virtual machines and for the operation of the cluster.

After installing Virtuozzo, you need to create a network bridge and add a real network interface there . This is necessary for your virtual machines to work with a shared network.

prlsrvctl net add network1
prlsrvctl net set network1 -t bridged --ifname enp1s0d1 #enp1s0d1 имя реальной сетевой

Disks

Do not add disks to Storage during the installation of the hypervisor . Rather, one will still have to, but after installation it will need to be removed and added again. The thing is that by default Virtuozzo included all disks in the tier0 group. This is the default cluster group of disks; they are also the slowest. To split the types of disks in the cluster into ssd and not ssd, you will have to unload all the disks that have already been added and add them again to the desired tier. Changing the tier of a disk that is already in the cluster is not possible. Even if you do not plan to have such a separation today, just do it now. It will not be worse, but in the future get rid of many unnecessary difficulties.

Another problem is that when you delete disks according to instructions, there are tails in the form of services that try to start CS for a disk that is no longer there. And if you deleted and added the same disk 2 times? Then, at any restart of the service, the CS-disk will try to start twice. Here is a simple recipe on how to properly unload and remove CS without problems:

cs=1071 #где цифра - номер вашего диска в кластере
cluster=livelinux #имя вашего кластера
vstorage -c $cluster rm-cs --wait $cs
systemctl stop vstorage-csd.$cluster.$cs.service
systemctl disable vstorage-csd.$cluster.$cs.service
systemctl reset-failed vstorage-csd.$cluster.*.service
systemctl | grep vstorage-csd

To add a disk to the correct dash, you definitely need to prepare a completely empty disk with the command:

/usr/libexec/vstorage/prepare_vstorage_drive /dev/sdc --noboot

blkid #узнаем UUID диска
mkdir /vstorage/livelinux-cs4 #создаем папку для монтирования диска 
#добавляем автоматическое монтирование диска в /etc/fstab
UUID="3de71ff9-724f-483a-8984-3b78fdb3b327"  /vstorage/livelinux-cs4 ext4 defaults,lazytime   1 2
#монтируем новый диск
mount -all
#добавляем диск к кластеру livelinux (ваше имя) в тир 2 (самые быстрые диски)
vstorage -c livelinux make-cs -r /vstorage/livelinux-cs4/data -t 2
#чтобы добавить hdd-диск с локальным ssd-журналом, то команда будет такая 
#где -t 1 это тир hdd+ssd кеш, /vz/livelinux-cs6-sata-journal - путь к журналу на ssd, 
#-s 30240 это размер в мегабайтах.
vstorage -c livelinux make-cs -r /vstorage/livelinux-cs7-sata/data -t 1 -j /vz/livelinux-cs6-sata-journal -s 30240
#перезапускаем службу CS, чтобы кластер увидел ваш новый диск
systemctl restart vstorage-csd.target

In addition, Linux reserves some space on the disk for writing logs to the / home folder.
In our case, this is pointless, turn it off (in the latest versions of the cluster, this is done when preparing the disks, but it is better to verify for yourself)

tune2fs -m 0 /dev/sdc1

Memory

Imagine the situation: the server is a hypervisor for virtual machines and contains disk chunks (CS) for Virtuozzo Storage. As a result, Linux diligently puts all the data on physical disks in the disk cache of RAM and in every possible way strives to occupy it completely in order to speed up the system’s work with disks. CS (disks) and MDS (distributed file system) services cannot quickly get free pages of memory and are forced to try to free the cache from the cache every time, which is a long time. Plus, old editions of Virtuozzo Cluster could not work with memory if it was fragmented, i.e. they demanded a whole block of free memory in RAM. The result is simple - CS chunk services are falling, non-replicated copies of data are being formed in the cluster, the cluster is starting to replicate them, disk usage is increasing, CS are falling even more often,

We begin to understand and it turns out that, among other things, by default Virtuozzo includes a memory management mode that does not involve any overcommit at all. In other words, virtual machines expect that memory will be free for their work, and that no disk cache will interfere. But that is not all. By default, containers can go beyond their own disk cache to the general system disk cache of the hypervisor. In this case, as a rule, the system cache is already filled with the CS data cache.

I bring a number of recommendations that need to be implemented immediately, even before the first virtual machine is launched on the cluster:

#запрещаем контейнерам выходить за пределы своей собственной оперативной памяти для размещения дискового кеша
echo "PAGECACHE_ISOLATION=\"yes\"" >> /etc/vz/vz.conf
#cами службы CS требуют память для своей работы, особенно если есть журнал для hdd-дисков, тут мы делаем ограничение, теперь службы CS могут выедать для своей работы максимум 32ГБ
cat > /etc/vz/vstorage-limits.conf << EOL
{
    "VStorage": {
        "Path": "vstorage.slice/vstorage-services.slice",
        "Limit": {
            "Max": 34359738368
            "Min": 0,
            "Share": 0.7
        },
        "Guarantee": {
            "Max": 4294967296,
            "Min": 0,
            "Share": 0.25
        },
        "Swap": {
            "Max": 0,
            "Min": 0,
            "Share": 0
        }
    }
}
EOL
service vcmmd restart
#разрешаем оверкомит памяти, иначе дисковый кеш мешает работе виртуальных машин
prlsrvctl set --vcmmd-policy density
prlsrvctl info | grep "policy"

In addition, you can make some changes to the Linux settings in terms of working with memory:

touch /etc/sysctl.d/00-system.conf
echo "vm.min_free_kbytes=1048576" >> /etc/sysctl.d/00-system.conf
echo "vm.overcommit_memory=1" >> /etc/sysctl.d/00-system.conf
echo "vm.swappiness=10" >> /etc/sysctl.d/00-system.conf
echo "vm.vfs_cache_pressure=1000" >> /etc/sysctl.d/00-system.conf

sysctl -w vm.swappiness=10
sysctl -w vm.overcommit_memory=1
sysctl -w vm.min_free_kbytes=1048576
sysctl -w vm.vfs_cache_pressure=1000

And periodically flush the disk cache:

sync && echo 3 > /proc/sys/vm/drop_caches

Flushing the cache allows you to temporarily solve the above problems, but you need to do this carefully. Unfortunately, this has a bad effect on the performance of the disk subsystem, because instead of the cache, data begins to be read from disks again.

This is what the normal memory mode of Virtuozzo Storage looks like.



As you can see, the disk cache is stable, the server has enough completely free memory. Of course, there can be no talk of any real memory overcomit. You must consider this.

Free up disk space, clean up garbage

Containers and virtual machines in a cluster are stored as large flat files. When something is deleted inside the container or the virtual machine, in fact, the place does not become free, just the data blocks in the cluster are reset. In order to solve this problem, Virtuozzo developers have written the pcompact tool. This utility is launched by Cron on all servers at 3 a.m. at the same time and tries to defragment the images of virtual machines, as well as delete and clear unallocated memory pages in the cluster (i.e. return free space). The operation itself creates a very high load on the network, on disks, and also requires additional RAM for its work (sometimes quite a lot). This can lead to a decrease in cluster performance during this operation due to the high utilization of the network and disks.

Those. we have top-end network and disk utilization. Plus, if there is not enough free memory, CS chunks begin to fall, and the replication of missing data blocks begins. As a result, avalanche problems occur on all nodes, on all virtual machines, and so on.

In addition, by default, this operation lasts only 2 hours, after which it automatically stops. This means that if you have a dozen very large virtual machines, there is a chance that garbage cleaning will never reach all of them, as it will not fit into a two-hour timeout.

To reduce the negative impact of this process, we transferred this task from the krona to anacron. Then the procedure starts at any time at night and does not interrupt until complete completion. As a result, as a rule, the process works only on one cluster node at a time, which generally reduces the load and the risk of cluster degradation at night.

Monitoring

The cluster has good proprietary monitoring tools, such as:

vstorage stat --cluster livelinux #ваше имя кластера

vstorage top --cluster livelinux #ваше имя кластера

It is also possible to obtain data for zabbiksa and build all the necessary graphs right in it.

On the example of our internal company cloud :

Free space in different ranges and total licensed cluster location Cluster



status, how many percent are rebalanced, how many chunks are urgently replicated. Perhaps one of the most important schedules.



The total network load of the disks of all virtual machines in the cluster and the speed of replication and rebalancing.



The average and maximum disk queue in the cluster.



General statistics on the IOPS cluster.



In addition, do not forget that by default, the firewall on Virtuozzo prohibits the operation of everything that is clearly not allowed. For zabbix-agent, you need a pair of firewall enable rules:

firewall-cmd --permanent --add-port=10050/tcp
firewall-cmd --permanent --add-port=10051/tcp
firewall-cmd --permanent --add-port=5001/tcp
systemctl restart firewalld

And finally, a few useful commands for working with the cluster

Find out which virtual machine lives in a disk dash:

(for i in `ls /vstorage/livelinux/private`; do grep HOSTNAME /vstorage/livelinux/private/$i/ve.conf; vstorage -c livelinux get-attr /vstorage/livelinux/private/$i|grep tier; done) | (grep "HOSTNAME\|tier" | tr '\n  ' ' ' | tr 'H' '\n' | sed 's#tier=2#SSD#g' | sed 's#tier=1#HDD#g' | sed 's#tier=2#SSD#g' | sed  's#OSTNAME=##g' | sed 's#"##g') | awk '{print $2" "$1}' | sort

Manual start of the pcompact process

/usr/sbin/pcompact -v

By default, Virtuozzo Storage reserves some disk space, just in case. If you strongly pressed, then you can temporarily lower the percentage of reservation:

vstorage -c livelinux set-config mds.alloc.fill_margin=1

If you need to unmount the cluster file system, but do not want to overload the server, you need to turn off all virtual machines and run:

fusermount -uz /vstorage/livelinux #имя вашего кластера

Find out the statistics of the local data cache

watch cat /vstorage/livelinux/.vstorage.info/read_cache_info

Find out the status of your licenses

pstorage -c livelinux view-license #ваше имя кластера
vzlicupdate
vzlicview

Find out your hwid

pstorage -c livelinux stat --license-hwid

Sometimes it happens that the size of the pfcache log does not fit in the virtual disk allocated to it. There is the following solution

prl_disk_tool resize  resize --size 15G  --hdd /vz/pfcache.hdd/DiskDescriptor.xml

Remove MDS service from server

vstorage -c livelinux rm-mds 11 #11-id вашего mds

Start MDS on the local server

vstorage -c livelinux make-mds -a 172.17.0.254:2510 #локальный адрес на сервере -r /vz/mds/data #пусть для журналу mds, на ssd диске -b 172.17.0.255 -b 172.17.0.249 -b 172.17.0.248 -b 172.17.0.4 #адреса всех mds
systemctl restart vstorage-mdsd.target
systemctl enable vstorage-mdsd.target

Change the default number of copies of virtual machine data on cluster servers

vstorage set-attr -R /vstorage/livelinux replicas=2

Change the number of copies of data for a specific virtual machine

vstorage set-attr -R /vstorage/livelinux/private/a0327669-855d-4523-96b7-cf4d41ccff7e replicas=1

Change the tier of all virtual machines, as well as set the tier by default

vstorage set-attr -R /vstorage/livelinux/private tier=2
vstorage set-attr -R  /vstorage/livelinux/vmprivate tier=2

Change the tier of a specific virtual machine

vstorage set-attr -R /vstorage/livelinux/private/8a4c1475-4335-40e7-8c95-0a3a782719b1 tier=2

Stop and migrate all virtual machines on the hypervisor

hp=msk-07
for vm in `prlctl list | grep -v UUID | awk '{print $5}'`; do
prlctl stop $vm;
prlctl migrate $vm $hp;
ssh $hp -C prlctl start $vm; done

Force the virtual machine to start on another server , even if it was already running somewhere, but completely hung or was not correctly migrated:

prlctl register /vz/private/20f36a7f-f64d-46fa-b0ef-85182bc41433 --preserve-uuid --force

Listing commands to create and configure a container

prlctl create test.local --ostemplate centos-7-x86_64 --vmtype ct
prlctl set test.local  --hostname test.local  --cpus 4 --memsize 10G --swappages 512M --onboot yes
prlctl set test.local  --device-set hdd0 --size 130G
prlctl set test.local  --netif_add netif1
prlctl set test.local  --ifname netif1 --network network1
prlctl set test.local --ifname netif1 --ipadd 172.17.0.244/24
prlctl set test.local  --ifname netif1 --nameserver 172.17.0.1
prlctl set test.local  --ifname netif1 --gw 172.17.0.1
prlctl start test.local
prlctl enter test.local

Thank you very much for your attention.