Cluster Proxmox nodes using OpenVPN

Using the Proxmox virtualization environment, namely OpenVZ containers, to create shared hosting is no news to anyone. The server rented at the Hetzner site has been successfully managing its responsibilities for quite some time.

But time passed, the amount of data increased, clients multiplied, LA grew ... A new server was rented, Proxmox installed and configured, the administrator is eager to set up a cluster to migrate loaded containers to the new server. Instructions pools were found on google, and the wiki on the Proxmox project itself has the necessary information.

Servers are on different subnets. Proxmox uses corosync cluster nodes to synchronize cluster node settings. When adding a node to the cluster - an error:

Waiting for quorum ... Timed-out waiting for cluster [FAILED]

Admin in panic


Configure synchronization of Proxmox nodes located in any data center and having an external IP address. Organize a “cluster” in the understanding of Proxmox.


So what we have:

  • Proxmox 3.3, the “free” repository,
  • Server No.1:
    • dns:
    • name: node1
    • external ip: 144.76.ab

  • Server node number 2:
    • dns:
    • name: node2
    • external ip:

  • Cluster:
    • name: cluster

  • All hosting containers work on internal subnets of nodes. Cheap, angry, no comment.

We find out that synchronization does not work due to the fact that multicast requests, although they are sent, are cut with equipment. Nodes just do not see each other. Also, for synchronization try to use the IP addresses of available network interfaces. Those. either an external IP or IP subnet for the VM.


We will force multicast requests sent by corosync to go inside the same network for all nodes of the cluster. Let's raise our private subnet with OpenVPN and routing.

0. Cleansing

First you need to roll back all the changes made by an unsuccessful attempt to add a node to the cluster. It is assumed that nothing was configured on node2 yet, and there was no VM.

  • on node1:

    pvecm nodes
    service pve-cluster restart
    pvecm expected 1
    pvecm delnode node2
    pvecm cman restart

  • on node2:

    service pve-cluster stop
    service cman stop
    rm /etc/cluster/cluster.conf
    rm -rf /var/lib/pve-cluster
    rm -rf /var/lib/corosync
    service pve-cluster start
    service cman start

1. Network settings within the cluster

For some unification of the settings, we will coordinate the following parameters for the networks inside our future cluster:

  • OpenVPN subnet: will be
  • The node on which the OpenVPN server will run will be called “master”.
  • The subnet for containers on the nodes will be of the form: 10. [1-254] .0.0 / 16, where the second octet is the node number.
  • Suppose we have VMs with system services, for example, database servers.
    I assume in advance that a name server is configured on "master", with a zone, for example, ".hosting.lan".
    This will facilitate the transfer of VMs between nodes. Just change the internal IP after the transfer.
  • We configure network interfaces on Proxmox nodes accordingly. Correct, if necessary, the settings of the VM.

2. Set up the "master" node

2.1 OpenVPN

I will not go into the configuration of OpenVPN much, because many articles have been written. Including on the Habré . I will describe only the main features and settings:

  1. Install:

    apt-get install openvpn

  2. We create the file with the settings /etc/openvpn/node1.conf and allow the launch for it in / etc / default / openvpn

  3. In the settings file, enter the following parameters:

    # Для работы мульткаста используем tap
    dev                 tap
    proto               udp
    # Сделаем буфер UDP побольше
    sndbuf 393216
    rcvbuf 393216
    # Подсеть сервера
    # Пробросим мультакаст-запросы на подсеть этого узла
    # corosync иногда любит использовать адрес vmbr0
    # Пробросим трафик до подсетей узлов через VPN
    # и так для каждого нового узла...
    # Настройки для клиентов-узлов
    client-config-dir   clients

  4. In the / etc / openvpn / clients directory, we create files for client-node settings:

    # На узле 1 — обычные ВМ
    push "route"
    # А, например, на узле 3 — системные ВМ
    # push "route"
    # multicast — через VPN на master-узел
    push "route"
    push "dhcp-option DNS"
    push "dhcp-option DOMAIN hosting.lan"
    push "sndbuf 393216"
    push "rcvbuf 393216"
    # Для tap-устройства — IP + NetMask

  5. We start vpn:

    service openvpn restart

  6. Go to the connected node “node2”, install openvpn too, write the “master” file in / etc / default / openvpn.

    You will also need to install the resolvconf package. Unlike the master node. Otherwise, magic with domains for the internal network may not work. I also had to copy the original file to tail inside the /etc/resolvconf/resolv.conf.d/ directory. Otherwise, the name servers from hether were lost.

    Depending on the server settings, create a settings file for the client, which should contain the following parameters:

    dev tap
    proto udp
    remote <внешний IP или домен master>

  7. We start vpn:

    service openvpn restart

2.2 Host and service settings for the cluster

  1. On each node, you need to edit the / etc / hosts file and look as follows:
    # IPv4 localhost.localdomain localhost
    # external address and domain of the host
    # IPv6
    :: 1 ip6-localhost ip6- loopback
    fe00 :: 0 ip6-localnet
    ff00 :: 0 ip6-mcastprefix
    ff02 :: 1 ip6-allnodes
    ff02 :: 2 ip6-allrouters
    ff02 :: 3 ip6-allhosts

    xxxx: xxx: xxx: xxxx :: 2 ipv6.node1. ipv6.node1

    VPN # node1 master cluster node2

    # and so for each new node ...

    By specifying separately the IP addresses from the VPN subnet for the nodes, we force their use, because Proxmox services use short domain names.

  2. On the "master" we edit the file /etc/pve/cluster.conf, add the line multicast:

    If the file cannot be saved, then try restarting the service:

    cd /etc
    service pve-cluster restart

    and try to edit again.
    After editing:

    cd /etc
    service pve-cluster restart
    service cman restart

  3. Check the status of "master":

    pvecm status

    As a result, the following should be visible:
    Node ID: 1
    Multicast addresses:
    Node addresses:

3. Add the node to the cluster

These settings should already be enough for the cluster to work. Add the node to the cluster according to the instructions from the wiki:

  1. We pass to the node "node2"
  2. Enter:

    pvecm add master

    We answer questions, wait. We see that the quorum is reached.

    pvecm status

    Node ID: 2
    Multicast addresses:
    Node addresses:



  • Proxmox sees the nodes in the cluster. In theory, you can organize a cluster of nodes located anywhere. It is necessary that the master node has an "external, white" IP address.
  • Settings are synchronized.
  • VMs migrate between nodes.
  • The speed between nodes and "master" can exceed 400Mbit if you enable compression in OpenVPN. Depends on the data and settings of the external network, of course.


Alas, not everything is as good as we would like.

  • Sometimes the quorum is violated, the settings cease to be saved. Restarting cluster services helps - pve-cluster, cman. It is not clear yet, these are corosync or openvpn problems. At these moments, it is very fun to migrate VMs.
  • A cluster is not exactly a cluster, is it? What happens if the master node shuts down? We also include the hard-coded IP addresses of the nodes in the VPN settings, hosts.
  • Virtual machine traffic between node2 and node3 will go through master via VPN. Such a scheme will be convenient only for the case when the main VMs are on the master, and the system VMs are on the additional nodes.

References - Installation and configuration guide for OpenVPN 2.x-cluster-running-over-an-openvpn-intranet

Also popular now: