An “ideal” cluster. Part 2.1: virtual cluster on hetzner
I want to note that this instruction was born in the process of examining various virtualization systems within the walls of Acronis .
Proxmox showed itself on the good side, perhaps our experience will be useful for solving your problems.
When renting another server in a data center, everyone thinks about the rationality of its use.
After all, it is no secret to anyone that a well-tuned server should not be too busy and there should be enough resources in it to do other work. In addition to the above, fault tolerance is important and therefore keeping multiple copies of the same server as a hot swap seems like a great idea.
To solve these problems and need virtualization.
Now I will tell you how to quickly make a single cluster of linux and windows based servers from one server.
In further articles, I will try to explain how to raise a secure web-cluster and use all the charms of modern virtualization technologies.
This guide will focus on the free Proxmox virtualization system, it is freely available, but requires a fee for support. We will try to do without the support and commercial repository of Proxmox. Here's what Wikipedia says about this product.
Proxmox Virtual Environment (Proxmox VE) is an open source virtualization system based on Debian GNU / Linux. Developed by the Austrian company Proxmox Server Solutions GmbH, sponsored by Internet Foundation Austria.
As a hypervisor uses KVM and OpenVZ. Accordingly, it is able to run any supported KVM OS (Linux, * BSD, Windows and others) with minimal loss of performance and Linux without loss.
Management of virtual machines and administration of the server itself is done through the web interface or through the standard Linux command line interface.
Many options are available for the created virtual machines: the hypervisor used, the type of storage (image file or LVM), the type of emulated disk subsystem (IDE, SCSI or VirtIO), the type of emulated network card, the number of available processors, and others.
- Simple management through the web interface;
- Real-time load monitoring
- Library of installation images (in local or remote storage);
- Connection to the “physical” console of guest systems directly from the browser (via VNC);
- Combining servers in a cluster with the possibility of live migration of virtual machines (without stopping the guest system);
- Quick deployment of guest systems from templates (available only for OpenVZ);
- Automatic backup of virtual machines.
First of all, you need to order a server with debian 7 64 on board, the more memory the better! worry about the safety of your data, RAID 1 will not be redundant at all, although in itself it carries a number of risks. We are optimists, we take with RAID1.
As soon as we have root access to our new server, we get to work:
# Before installing proxmox itself, you need to decide on hostname and specify it
127.0.0.1 localhost x.x.x.x test.xxxx.info test # # IPv6 ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts x:x:x:4105::2 test.xxxx.info
# Change the time zone
echo "Europe/Moscow" > /etc/timezone dpkg-reconfigure -f noninteractive tzdata
# Create a folder for repositories
mkdir -p /etc/apt/sources.list.d/
# Download repositories
cd /etc/apt/sources.list.d/ wget http://sycraft.info/share/debian7/sources.list.d/debian7.list wget http://sycraft.info/share/debian7/sources.list.d/dotdeb7.list wget http://sycraft.info/share/debian7/sources.list.d/neurodebian.sources7.list wget http://sycraft.info/share/debian7/sources.list.d/proxmox7.list
# Install the keys
cd /root/ wget http://www.dotdeb.org/dotdeb.gpg cat dotdeb.gpg | apt-key add - apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A040830F7FAC5991 apt-key adv --recv-keys --keyserver pgp.mit.edu 2649A5A9 wget -O- "http://download.proxmox.com/debian/key.asc" | apt-key add - rm *.gpg
# Update system
apt-get update && apt-get upgrade -f -y
# We set the necessary minimum
apt-get install ntp screen mc git ntpdate sudo zip unzip pigz locales tzdata nano aptitude htop iotop sysstat rkhunter chkrootkit nscd lsof strace subversion multitail -y -f
# Install the kernel from proxmox
apt-get install pve-firmware pve-kernel-2.6.32-26-pve -y -f apt-get install pve-headers-2.6.32-26-pve -y -f
# Cleaning the system from old kernels
apt-get remove linux-image-amd64 linux-image-3.2.0-4-amd64 -y -f
# Generation grub
# We were lucky, our server booted up and now you can install proxmox yourself
apt-get install proxmox-ve-2.6.32 ntp ssh lvm2 postfix ksm-control-daemon vzprocps open-iscsi bootlogd -y
# Delete repository for paid proxmox
rm -fr /etc/apt/sources.list.d/pve-enterprise.list
# Add iptables modules for all occasions
IPTABLES="ipt_owner ipt_REDIRECT ipt_recent ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ipt_state iptable_nat ip_nat_ftp"
# Add modules when loading the kernel (an extra step, but just in case)
ipt_MASQUERADE ipt_helper ipt_REDIRECT ipt_state ipt_TCPMSS ipt_LOG ipt_TOS tun iptable_nat ipt_length ipt_tcpmss iptable_mangle ipt_limit ipt_tos iptable_filter ipt_helper ipt_tos ipt_ttl ipt_REJECT loop
Then a few words about the proposed architecture:
- We order together with the server 2 external (public) IP addresses, on the first is the service port of the web panel proxmox, ssh, mysql and other service ports that no one should know about
- The second address serves ports that should be accessible to everyone. For example 80 and 443 and all. In addition, this address was raised on an empty virtual machine devoid of unnecessary services. The rest will be resolved by port forwarding.
# Save current iptables rules
iptables-save > /etc/iptables.up.rules
# Add the rules to the * nat section for our external business address
*nat :PREROUTING ACCEPT [2164:136969] :POSTROUTING ACCEPT [58:3659] :OUTPUT ACCEPT [0:0] -A PREROUTING -d x.x.16.182/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 192.168.8.2:22 -A POSTROUTING -o vmbr0 -j MASQUERADE -A POSTROUTING -d x.x.16.182 -p tcp -s 192.168.8.0/24 --dport 22 -j SNAT --to-source x.x.16.182 COMMIT
# Check the rules, there should be no errors
iptables-restore < /etc/iptables.up.rules
A very important rule is POSTROUTING. If you want inside one of the virtual machines to access the forwarded port of the external address, nothing will work without this rule!
# Download openvz container images
cd /var/lib/vz/template/cache/ wget http://download.openvz.org/template/precreated/debian-7.0-x86_64.tar.gz wget http://download.openvz.org/template/precreated/centos-6-x86_64.tar.gz wget http://download.openvz.org/template/precreated/ubuntu-13.10-x86_64.tar.gz
# Drivers in case we need windows
cd /var/lib/vz/template/iso/ wget http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/virtio-win-0.1-74.iso
# Next, go to the external address of our server xx16.182 : 8006 /
After authorization, we see a message about using the free version, we need it. Just getting used to clicking OK or buying a subscription
Network Setup for Hetznet
# Let's start reconfiguring the network, setting up network bridges may seem strange, but in Hetzner there is a limit on the number of MAC addresses on the switch port, all external addresses will be provided by 1 MAC.
This setting works equally well in DCs without such restrictions, just a universal option
There is also a private network 192.168.8.0/16 - we use it for the internal network between our virtual machines
# Next, we reboot our server and look at our network settings
auto lo iface lo inet loopback auto eth0 iface eth0 inet static address x.x.16.182 netmask 255.255.255.224 pointopoint x.x.16.129 gateway x.x.16.129 dns-nameservers 220.127.116.11 auto vmbr0 iface vmbr0 inet static address x.x.16.182 netmask 255.255.255.224 bridge_ports none bridge_stp off bridge_fd 0 pre-up iptables-restore < /etc/iptables.up.rules up ip route add x.x.150/32 dev vmbr0 # auto vmbr1 iface vmbr1 inet static address 192.168.8.100 netmask 255.255.0.0 bridge_ports none bridge_stp off bridge_fd 0
# We write the gateway of the external service IP in pointopoint and gateway, in vmbr0 the same addresses are indicated but without gateway, the up ip route add route and pre-up iptables-restore firewall rules must be registered to the second address on which the public ports will be
For general development, here is an example of network settings for DCs without restrictions on the number of MAC addresses
nano /etc/network/interfaces # network interface settings auto lo iface lo inet loopback auto eth0 iface eth0 inet manual auto vmbr0 iface vmbr0 inet static address x.x.16.182 netmask 255.255.255.0 gateway x.x.16.1 dns-nameservers 18.104.22.168 bridge_ports eth0 bridge_stp off bridge_fd 0 pre-up iptables-restore < /etc/iptables.up.rules auto vmbr1 iface vmbr1 inet static address 192.168.8.100 netmask 255.255.0.0 bridge_ports none bridge_stp off bridge_fd 0
# Here is an example of how we can install our Windows (if necessary)
In the settings of the video card, I specify SPICE and here is the client for it www.spice-space.org/download.html
Network and disk - virtio, I immediately install the drivers second cd-rom for the downloaded virtio iso
Well and the last in this article is the gw configuration. virtual machine that will forward us public ports. There will be no SSH or other services listening on the network on this virtual machine - it is a firewall node.
You create a CT with the debian image with the Network Device network.
In the container itself, it will look like this:
auto lo iface lo inet loopback auto eth0 iface eth0 inet static address x.x.x.150 netmask 255.255.255.255 pointopoint x.x.16.182 gateway x.x.16.182 pre-up iptables-restore < /etc/iptables.up.rules auto eth1 iface eth1 inet static address 192.168.8.1 netmask 255.255.0.0
# Pay attention to the mask, gateway and pointopoint for this interface - the address of our public service network.
# Add the rules to the * nat section for our external public address
*nat :PREROUTING ACCEPT [2164:136969] :POSTROUTING ACCEPT [58:3659] :OUTPUT ACCEPT [0:0] -A PREROUTING -d x.x.x.150/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.8.5:22 -A POSTROUTING -o eth0 -j MASQUERADE -A POSTROUTING -d x.x.x.150 -p tcp -s 192.168.8.0/24 --dport 80 -j SNAT --to-source x.x.x.150 COMMIT
# Allow probross traffic when masquerading
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf sysctl -p
# In this container, delete all unnecessary!
apt-get purge -y -f openssh-server postfix ssh samba bind9 sendmail apache2*apt-get autoremove -y
If you have any difficulties or need a special person to make the story come true - I will always be happy to help! my contacts are welcome
In continuation of the topic, my article “Ideal” www cluster. Part 1. Frontend: NGINX + Keepalived (vrrp) on CentOS.
There will be, I hope, many, many more articles on this topic! Thanks for attention