Expanding LINSTOR for Proxmox

Original author: kvaps
  • Transfer
  • Tutorial

image


LINBIT recently released its new solution for orchestrating and managing multiple DRBD arrays.


For example, you can have several nodes and each will have its own LVM or ZFS pool in which LINSTOR will automatically create new volumes and replicate them between nodes using the DRBD protocol.


LINSTOR supports thin-provisioning, snapshots, and many other interesting pieces.
This solution is well suited for virtual machines and containers.


LINSTOR Satellites


I believe that you already have a Proxmox cluster, I have three nodes pve1, pve2and pve3.


So the first step is to install drbd-utils and build a kernel module on each node.


Add a linstor repository and install kernel-headers , as well as all the necessary packages from it:


wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add - 
echo"deb http://packages.linbit.com/proxmox/ proxmox-5 drbd-9.0" \
  > /etc/apt/sources.list.d/linbit.list
apt-get update
apt-get -y install pve-headers
apt-get -y install drbd-dkms drbdtop

After installation, let's check your version of the kernel module:


modprobe drbd
cat /proc/drbd

If you see version 8, then something went wrong and you have loaded the in-tree kernel module, check dkms statusto learn more details about building the dkms module directly.
For LINSTOR, you must use DRBD version 9.


Next, on each node, install the linstor-proxmox and linstor-satellite packages :


apt-get -y install linstor-proxmox linstor-satellite
systemctl start linstor-satellite.service
systemctl enable linstor-satellite.service

LINSTOR Controller


We will deploy the controller inside the LXC container.


Download the debian template:


wget http://download.proxmox.com/images/system/debian-9.0-standard_9.3-1_amd64.tar.gz -P /var/lib/vz/template/cache/

Now we will create a container for the controller:


pct create 100 local:vztmpl/debian-9.0-standard_9.3-1_amd64.tar.gz \
  --hostname=linstor-controller \
  --net0=name=eth0,bridge=vmbr0,gw=10.1.0.1,ip=10.1.0.123/16

Run the container and enter it:


pct start 100
pct exec 100 bash

Install updates:


apt-get update 
apt-get -y upgrade

Add a linstor repository and install linstor-controller and linstor-client :


wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add - 
echo"deb http://packages.linbit.com/proxmox/ proxmox-5 drbd-9.0" \
  > /etc/apt/sources.list.d/linbit.list
apt-get update && apt-get install -y linstor-controller linstor-client
systemctl start linstor-controller.service
systemctl enable linstor-controller.service

Linstor requires a customized locale. Set up the locale:


sed -i '/en_US.UTF-8 UTF-8/ s/^# //' /etc/locale.gen
locale-gen

Let's set up the time zone right away:


dpkg-reconfigure tzdata

Configuring Storage


Create nodes:


linstor node create pve1 10.1.0.11
linstor node create pve2 10.1.0.12
linstor node create pve3 10.1.0.13

For each node, we describe an additional interface that will be used for DRBD replication:


linstor node interface create pve1 data 10.2.0.11
linstor node interface create pve2 data 10.2.0.12
linstor node interface create pve3 data 10.2.0.13

Example command output linstor node list:


╭──────────────────────────────────────────────────────────────╮
┊ Node ┊ NodeType  ┊ Addresses                        ┊ State  ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ pve1 ┊ SATELLITE ┊ 10.0.1.11,10.0.2.11:3366 (PLAIN) ┊ Online ┊
┊ pve2 ┊ SATELLITE ┊ 10.0.1.12,10.0.2.12:3366 (PLAIN) ┊ Online ┊
┊ pve3 ┊ SATELLITE ┊ 10.0.1.13,10.0.2.13:3366 (PLAIN) ┊ Online ┊
╰──────────────────────────────────────────────────────────────╯

Create pools:


On each node you should prepare lvm thin-pool for drbd:


lvcreate -L 800G --thinpool drbdpool pve

Now you can add them to linstor:


linstor storage-pool create lvmthin pve1 drbdpool pve/drbdpool
linstor storage-pool create lvmthin pve2 drbdpool pve/drbdpool
linstor storage-pool create lvmthin pve3 drbdpool pve/drbdpool

Set up pools to use the datainterface:


linstor storage-pool set-property pve1 drbdpool PrefNic data
linstor storage-pool set-property pve2 drbdpool PrefNic data
linstor storage-pool set-property pve3 drbdpool PrefNic data

Example command output linstor storage-pool list:


╭───────────────────────────────────────────────────────────────────────────────────────────
┊ StoragePool ┊ Node ┊ Driver        ┊ PoolName     ┊ FreeCapacity ┊ TotalCapacity ┊ Support
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
┊ drbdpool    ┊ pve1 ┊ LvmThinDriver ┊ hv1/drbdpool ┊      800 GiB ┊       800 GiB ┊ true              
┊ drbdpool    ┊ pve1 ┊ LvmThinDriver ┊ hv1/drbdpool ┊      800 GiB ┊       800 GiB ┊ true              
┊ drbdpool    ┊ pve3 ┊ LvmThinDriver ┊ hv3/drbdpool ┊      800 GiB ┊       800 GiB ┊ true              
╰───────────────────────────────────────────────────────────────────────────────────────────

Now we can add our new repository to the Proxmox config:


cat >> /etc/pve/storage.cfg <<EOF
drbd: drbdpool
        content rootdir,images
        controller 10.1.0.123
        controllervm 100
        redundancy 3
EOF

Configuring HA for the controller


Now we will transfer our linstor container to linstor storage.


First we need to create a virtual disk for it:


pct exec 100 bash
linstor resource-definition create vm-100-disk-0
linstor volume-definition create vm-100-disk-0 4G
linstor resource create vm-100-disk-0 --auto-place 3 -s pve

Example command output linstor resource list:


╭──────────────────────────────────────────╮
┊ ResourceName    ┊ Node ┊ Port ┊    State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ vm-100-disk-0   ┊ pve1 ┊ 7000 ┊ UpToDate ┊
┊ vm-100-disk-0   ┊ pve2 ┊ 7000 ┊ UpToDate ┊
┊ vm-100-disk-0   ┊ pve3 ┊ 7000 ┊ UpToDate ┊
╰──────────────────────────────────────────╯

By default, linstor-satellite deletes the configs of all resources at the time the service starts, then it expects a new configuration from the linstor-controller , but if we keep the linstor-controller in the same place as the data, we must make an exception for its resource by adding the option --keep-res=vm-100for linstor-satellite on all nodes:


echo -e "[Service]\nExecStart=\nExecStart=/usr/share/linstor-server/bin/Satellite --logs=/var/log/linstor-satellite --config-directory=/etc/linstor --keep-res=vm-100" | SYSTEMD_EDITOR=tee systemctl edit linstor-satellite.service

This action is to prevent the resource from being deleted vm-100-disk-0every time it is started.


Now we stop the linstor-controller container and transfer all data from the local disk to the drbd disk.


pct shutdown 100
dd if=/var/lib/vz/images/100/vm-100-disk-0.raw of=/dev/drbd/by-res/vm-100-disk-0/0 bs=8M status=progress
e2fsck -f /dev/drbd/by-res/vm-100-disk-0/0 
resize2fs /dev/drbd/by-res/vm-100-disk-0/0

Update the container config:


sed -i '/^rootfs:/d' /etc/pve/lxc/100.conf
echo'rootfs: drbdpool:vm-100-disk-0,size=4G' >> /etc/pve/lxc/100.conf

Run the container, and if everything is OK, remove the old disk:


pct start 100
rm -f /var/lib/vz/images/100/vm-100-disk-0.raw

Now we just need to add our container to proxmox ha-manager :


ha-manager add ct:100 --max_relocate=3 --max_restart=3

PROFIT


Performance and tuning


At this point, to change the global parameters of the controller, you will need to disable authentication on the controller:



To automatically resolve split-brain situations, add the following options for the controller:


linstor controller drbd-options \
  --after-sb-0pri=discard-zero-changes \
  --after-sb-1pri=discard-secondary \
  --after-sb-2pri=disconnect

For my 10G network, I found the following settings to be optimal for fast synchronization:


linstor controller drbd-options \
  --max-buffers=36864 \
  --rcvbuf-size=2097152 \
  --sndbuf-size=1048576
linstor controller drbd-options \
  --c-fill-target=10240 \
  --c-max-rate=737280 \
  --c-min-rate=20480 \
  --c-plan-ahead=10
Tags:

Also popular now: