February 26, 2015 at 10:27

Cluster Proxmox nodes using OpenVPN

Using the Proxmox virtualization environment, namely OpenVZ containers, to create shared hosting is no news to anyone. The server rented at the Hetzner site has been successfully managing its responsibilities for quite some time.

But time passed, the amount of data increased, clients multiplied, LA grew ... A new server was rented, Proxmox installed and configured, the administrator is eager to set up a cluster to migrate loaded containers to the new server. Instructions pools were found on google, and the wiki on the Proxmox project itself has the necessary information.

Servers are on different subnets. Proxmox uses corosync cluster nodes to synchronize cluster node settings. When adding a node to the cluster - an error:

Waiting for quorum ... Timed-out waiting for cluster [FAILED]

Admin in panic


Task:


Configure synchronization of Proxmox nodes located in any data center and having an external IP address. Organize a “cluster” in the understanding of Proxmox.

Given:


So what we have:

  • Proxmox 3.3, the “free” repository,
  • Server No.1:
    • dns: node1.example.com
    • name: node1
    • external ip: 144.76.ab

  • Server node number 2:
    • dns: node2.example.com
    • name: node2
    • external ip: 144.76.cd

  • Cluster:
    • name: cluster

  • All hosting containers work on internal subnets of nodes. Cheap, angry, no comment.

We find out that synchronization does not work due to the fact that multicast requests, although they are sent, are cut with equipment. Nodes just do not see each other. Also, for synchronization try to use the IP addresses of available network interfaces. Those. either an external IP or IP subnet for the VM.

Decision:


We will force multicast requests sent by corosync to go inside the same network for all nodes of the cluster. Let's raise our private subnet with OpenVPN and routing.

0. Cleansing

First you need to roll back all the changes made by an unsuccessful attempt to add a node to the cluster. It is assumed that nothing was configured on node2 yet, and there was no VM.

  • on node1:

    pvecm nodes
service pve-cluster restart
pvecm expected 1
pvecm delnode node2
pvecm cman restart

  • on node2:

    service pve-cluster stop
service cman stop
rm /etc/cluster/cluster.conf
rm -rf /var/lib/pve-cluster
rm -rf /var/lib/corosync
service pve-cluster start
service cman start

1. Network settings within the cluster

For some unification of the settings, we will coordinate the following parameters for the networks inside our future cluster:

  • OpenVPN subnet: will be 10.0.0.0/24.
  • The node on which the OpenVPN server will run will be called “master”.
  • The subnet for containers on the nodes will be of the form: 10. [1-254] .0.0 / 16, where the second octet is the node number.
  • Suppose we have VMs with system services, for example, database servers.
    I assume in advance that a name server is configured on "master", with a zone, for example, ".hosting.lan".
    This will facilitate the transfer of VMs between nodes. Just change the internal IP after the transfer.
  • We configure network interfaces on Proxmox nodes accordingly. Correct, if necessary, the settings of the VM.

2. Set up the "master" node

2.1 OpenVPN

I will not go into the configuration of OpenVPN much, because many articles have been written. Including on the Habré . I will describe only the main features and settings:

  1. Install: 

    apt-get install openvpn

  2. We create the file with the settings /etc/openvpn/node1.conf and allow the launch for it in / etc / default / openvpn

  3. In the settings file, enter the following parameters:

    # Для работы мульткаста используем tap
dev                 tap
proto               udp
# Сделаем буфер UDP побольше
sndbuf 393216
rcvbuf 393216
# Подсеть сервера
server              10.0.0.0   255.255.255.0
# Пробросим мультакаст-запросы на подсеть этого узла
# corosync иногда любит использовать адрес vmbr0
route               224.0.0.0   240.0.0.0   10.1.0.1
# Пробросим трафик до подсетей узлов через VPN
route               10.2.0.0    255.255.255.0   10.0.0.2
route               10.3.0.0    255.255.255.0   10.0.0.3
# и так для каждого нового узла...
# Настройки для клиентов-узлов
client-config-dir   clients
client-to-client

  4. In the / etc / openvpn / clients directory, we create files for client-node settings:

    /etc/openvpn/clients/node2:
# На узле 1 — обычные ВМ
push "route 10.1.0.0 255.255.0.0"
# А, например, на узле 3 — системные ВМ
# push "route 10.3.0.0 255.255.0.0"
# multicast — через VPN на master-узел
push "route 224.0.0.0 240.0.0.0"
push "dhcp-option DNS 10.0.0.1"
push "dhcp-option DOMAIN hosting.lan"
push "sndbuf 393216"
push "rcvbuf 393216"
# Для tap-устройства — IP + NetMask
ifconfig-push 10.0.0.2 255.255.0.0

  5. We start vpn:

    service openvpn restart

  6. Go to the connected node “node2”, install openvpn too, write the “master” file in / etc / default / openvpn.

    You will also need to install the resolvconf package. Unlike the master node. Otherwise, magic with domains for the internal network may not work. I also had to copy the original file to tail inside the /etc/resolvconf/resolv.conf.d/ directory. Otherwise, the name servers from hether were lost.

    Depending on the server settings, create a settings file for the client, which should contain the following parameters:

    /etc/openvpn/master.conf:
client
dev tap
proto udp
remote <внешний IP или домен master>

  7. We start vpn: 

    service openvpn restart


2.2 Host and service settings for the cluster

  1. On each node, you need to edit the / etc / hosts file and look as follows:
    # IPv4
    127.0.0.1 localhost.localdomain localhost
    # external address and domain of the host
    144.76.ab node1.example.com
    #
    # IPv6
    :: 1 ip6-localhost ip6- loopback
    fe00 :: 0 ip6-localnet
    ff00 :: 0 ip6-mcastprefix
    ff02 :: 1 ip6-allnodes
    ff02 :: 2 ip6-allrouters
    ff02 :: 3 ip6-allhosts

    xxxx: xxx: xxx: xxxx :: 2 ipv6.node1. example.com ipv6.node1

    #
    VPN #

    10.0.0.1 node1 master cluster
    10.0.0.2 node2
    # and so for each new node ...

    By specifying separately the IP addresses from the VPN subnet for the nodes, we force their use, because Proxmox services use short domain names.

  2. On the "master" we edit the file /etc/pve/cluster.conf, add the line multicast:


    If the file cannot be saved, then try restarting the service:

    cd /etc
service pve-cluster restart

    and try to edit again.
    After editing:

    cd /etc
service pve-cluster restart
service cman restart

  3. Check the status of "master":

    pvecm status

    As a result, the following should be visible:
    ...
    Node ID: 1
    Multicast addresses: 224.0.2.1
    Node addresses: 10.0.0.1

3. Add the node to the cluster

These settings should already be enough for the cluster to work. Add the node to the cluster according to the instructions from the wiki:

  1. We pass to the node "node2"
  2. Enter:

    pvecm add master

    We answer questions, wait. We see that the quorum is reached.

    pvecm status

    ...
    Node ID: 2
    Multicast addresses: 224.0.2.1
    Node addresses: 10.0.0.2

Result


Positive

  • Proxmox sees the nodes in the cluster. In theory, you can organize a cluster of nodes located anywhere. It is necessary that the master node has an "external, white" IP address.
  • Settings are synchronized.
  • VMs migrate between nodes.
  • The speed between nodes and "master" can exceed 400Mbit if you enable compression in OpenVPN. Depends on the data and settings of the external network, of course.


Negative

Alas, not everything is as good as we would like.

  • Sometimes the quorum is violated, the settings cease to be saved. Restarting cluster services helps - pve-cluster, cman. It is not clear yet, these are corosync or openvpn problems. At these moments, it is very fun to migrate VMs.
  • A cluster is not exactly a cluster, is it? What happens if the master node shuts down? We also include the hard-coded IP addresses of the nodes in the VPN settings, hosts.
  • Virtual machine traffic between node2 and node3 will go through master via VPN. Such a scheme will be convenient only for the case when the main VMs are on the master, and the system VMs are on the additional nodes.


References


habrahabr.ru/post/233971 - Installation and configuration guide for OpenVPN
pve.proxmox.com/wiki/Proxmox_VE_2.0_Cluster
pve.proxmox.com/wiki/Multicast_notes
www.nedproductions.biz/wiki/configuring-a-proxmox-ve- 2.x-cluster-running-over-an-openvpn-intranet
Tags:

Also popular now: