Migrating a physical machine in a ProxmoxVE2 (3) cluster from DRBD to new hardware

Having a cluster of two servers standing on Proxmox with DRBD-integrated storage, you may need to update the node in this cluster without stopping the machines in it. The task is not difficult, but some moments are not always remembered in the process.
Therefore, for myself, the actions were recorded for the future copy-paste in the console. Well, once written, why not share it with people?
PS In the end, a small bonus about the expansion of a block device called DRBD, tuning and useful links.


Let's get started.

1. We migrate all virtual machines to one node that remains alive (PM2).
2. We go the console on PM2 from the privileged user. We allow one node to be in the cluster:

pvecm expected 1

3. We look at the nodes in the cluster:

pvecm nodes

4. We remove the moving node from the cluster:

pvecm delnode pm1

5. Stop the DRBD synchronization:

drbdadm disconnect all

and check the status of StandAlone with the command

cat /proc/drbd

6. Delete entries about the old machine in / etc / pve / priv / {autorized_keys, know_hosts}, as well as in /root/.ssh
7. Install the ProxmoxVE2 on the new machine (PM1), indicating the sizes of the partitions during installation, writing in the invitation
boot: linux ext4 maxroot = 8 swapsize = 1 maxvz = 10 (there is also a key minfree = 200)
8. Go to the console with fresh PM1, update:

apt-get update && apt-get dist-upgrade

9. Add to the cluster:

pvecm add 192.168.0.12

(192.168.0.12 is the IP address of PM2)
10. Install the utilities for working with DRBD:

apt-get install drbd8-utils

but since in the debian repository old utilities we need to update them to similar ones in the kernel:

dpkg -i drbd8-utils_8.3.13-0_amd64.deb

(in Proxmox 3 based on Debian 7, the version is relevant, so skip this point)
11. Create / copy the file from PM2 /etc/drbd.d/r0.res for an example with the following contents:

resource r0 {
protocol C;
  startup {
    wfc-timeout  0;
    degr-wfc-timeout 600;
    become-primary-on both;
  }
  syncer {
    rate 500M;
  }
  net {
    cram-hmac-alg sha1;
    shared-secret "my-secret";
    allow-two-primaries;
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
    sndbuf-size 0;
    no-tcp-cork;
    unplug-watermark 16;
    max-buffers 8000;
    max-epoch-size 8000;
  }
  on pm1 {
    device /dev/drbd0;
    disk /dev/pve/drbd;
    address 192.168.0.11:7788;
    meta-disk internal;
  }
  on pm2 {
    device /dev/drbd0;
    disk /dev/pve/drbd;
    address 192.168.0.12:7788;
    meta-disk internal;
}

12. Create a logical partition exactly the same size as on a working machine:

lvcreate -L450G -n drbd0 pve

and see what we have done with the team

lvscan

13. We start drbd:

/etc/init.d/drbd start

14. Create a metadata:

drbdadm create-md r0

15. Just in case, make secondary

drbdadm secondary r0

and disconnect from all

drbdadm disconnect r0

16. We say that on our machine the storage is not up to date with the command:

drbdadm -- --discard-my-data connect r0

and reboot the machine
17. In the console of the first server, we look at the synchronization status with the command:

watch cat /proc/drbd

18. In the web interface in the cluster front, we allow our

Bonus nodes near the repository with drbd : Expanding the DRBD volume

This is all true for the external meta disk.
1. We added disk space in the reid. We expand the section on which drbd lies:

lvextend  -L+250G /dev/pve/drbd0

2. On an empty machine, make the storage secondary:

drbdadm secondary r0

3. We expand drbd by running on the machine where the virtual machines are spinning, because the disk on the second machine is temporarily locked:

drbdadm resize r0

4. The extension of the drbd device will increase only this block device, but not pv, to increase it:

pvresize /dev/pve/drbd0

If the internal disk, you will have to temporarily or permanently transfer the metadata to a separate disk. Without stopping, this can be done in two stages
1. On one of the servers:
a) Stop drbd:

drbdadm down r0

then we change the
meta-disk internal lines
to
meta-disk / dev / ”your metadata device” in the config [0]
for which the ssd disk is highly recommended, but for the time being you can create an lvm partition.
b) Then on another machine

drbdadm -- --overwrite-data-of-peer primary r0

but on the current

drbdadm up r0

and look at the synchronization process

watch cat /proc/drbd 

After the process is completed, on this server metadata is transferred to a separate device.
2. Repeat step 1 on another server.

If there is no ssd, then after expansion, in order not to spoil the speed of the storage, you will have to do the reverse manipulations. But it’s better to buy a couple of ssd, or maybe four, in reserve or by connecting them to RAID1 on each of the servers.

It’s quite natural to make backups before any actions, and the line in the /etc/vzdump.conf file will help to make them faster:

bwlimit: 100000

which will make a speed limit of 100 megabits, by default it costs only 10, which is quite normal for a 100 megabit network, but for gigabits (a) it is too soft.

Useful links:
Proxmox website Proxmox
wiki
Proxmox forum
Download image for

UPD installation : Congratulations on the release of Proxmox 3.0 based on Debian Wheezy