February 6, 2014 at 16:18Creation of reliable storage distributed to several servers via nfs
When creating a cluster for processing calls based on CGP , it became necessary to configure uninterrupted storage mounted from several servers.
Ubuntu Server 10.10 was taken as a distribution for the servers. The disk space was divided into two logical drives (sda1 for installing the system, and sda2 for the shared disk itself).
After installing the base system, you must additionally install the following packages: heartbeat, pacemaker, drbd8-utils, xfs, xfsprogs, nfs-kernel-server.
Heartbeat and pacemaker are needed for server clustering. The storage is based on drbd, xfs was used as a file system. Distribution of the file system to servers done by nfs.
For the nodes, the names u1 and u2 were chosen. For convenience, these names were immediately spelled out in / etc / hosts:
u0 is the address at which storage is available for mounting the file system from other servers.
The storage configuration file is located in /etc/drbd.d/r0.res:
The file is the same on both nodes, so you can create one and then copy it to the second.
Such a cruel setting for the treatment of split brain was chosen, since the storage is mainly used to store the system configuration. That is, the loss of recent changes is not as critical as the loss of calls during downtime due to slit blain.
After creating the configuration file, you need to create the disks themselves on both servers of the cluster:
After that, you can start drbd. It is important to run the drbd daemon on both servers with a difference of less than a minute (timeout for communication with other cluster members):
After this, in / proc / drbd there will be a state of storage, in something like this:
That is, two nodes are in secondary mode, while the disk is not consistent. In order to get out of this situation, it is necessary to forcibly declare one of the cluster nodes as the main one; to do this, execute the command:
After that, drbd will start updating the state of the secondary disk:
After the synchronization is over, you can create a file system on the server where drbd is in primary state:
For the next steps, the standard daemon launch mechanism will interfere with us. Therefore, you must run the command on both servers:
The configuration is created in the /etc/heartbeat/ha.cf file. It is the same on both nodes, so you can create it on one and then copy it to the second.
The second file is used to authenticate / etc / heartbeat / authkeys:
It’s better not to wait for a warning from heartbeat for incorrect file attributes and change them in advance:
After that, you can start heartbeat:
After a while, the crm_mod command should show that two nodes are connected to each other:
Next, run the crm configure edit command and enter the configuration for the cluster:
Exportfs settings indicate that the directory where the shared file system / shared will be mounted can be mounted by nfs to all servers on the 10.0.0.0 network.
After a while, the cluster monitor should show the status of the resources:
You can use / etc / fstab for this:
Ubuntu Server 10.10 was taken as a distribution for the servers. The disk space was divided into two logical drives (sda1 for installing the system, and sda2 for the shared disk itself).
After installing the base system, you must additionally install the following packages: heartbeat, pacemaker, drbd8-utils, xfs, xfsprogs, nfs-kernel-server.
Heartbeat and pacemaker are needed for server clustering. The storage is based on drbd, xfs was used as a file system. Distribution of the file system to servers done by nfs.
1. System setup
For the nodes, the names u1 and u2 were chosen. For convenience, these names were immediately spelled out in / etc / hosts:
10.0.0.84 u1 10.0.0.115 u2 10.0.0.120 u0
u0 is the address at which storage is available for mounting the file system from other servers.
2. Configuring drbd
The storage configuration file is located in /etc/drbd.d/r0.res:
resource r0 {
protocol C;
syncer {
rate 4M;
}
startup {
wfc-timeout 15;
degr-wfc-timeout 60;
}
net {
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
cram-hmac-alg sha1;
shared-secret somesecretword;
}
on u1 {
device / dev / drbd0;
disk / dev / sda2;
address 10.0.0.84:7788;
meta-disk internal;
}
on u2 {
device / dev / drbd0;
disk / dev / sda2;
address 10.0.0.115:7788;
meta-disk internal;
}
}
The file is the same on both nodes, so you can create one and then copy it to the second.
Such a cruel setting for the treatment of split brain was chosen, since the storage is mainly used to store the system configuration. That is, the loss of recent changes is not as critical as the loss of calls during downtime due to slit blain.
After creating the configuration file, you need to create the disks themselves on both servers of the cluster:
dd if = / dev / zero of = / dev / sda2 bs = 64M drbdadm create-md r0
After that, you can start drbd. It is important to run the drbd daemon on both servers with a difference of less than a minute (timeout for communication with other cluster members):
/etc/init.d/drbd start
After this, in / proc / drbd there will be a state of storage, in something like this:
0: cs: Connected ro: Secondary / Secondary ds: Inconsistent / Inconsistent C r ----
That is, two nodes are in secondary mode, while the disk is not consistent. In order to get out of this situation, it is necessary to forcibly declare one of the cluster nodes as the main one; to do this, execute the command:
drbdadm - --overwrite-data-of-peer primary r0
After that, drbd will start updating the state of the secondary disk:
0: cs: SyncSource ro: Primary / Secondary ds: UpToDate / Inconsistent C r ----
ns: 241984 nr: 0 dw: 0 dr: 242184 al: 0 bm: 14 lo: 510 pe: 179 ua: 510 ap: 0 ep: 1 wo: b oos: 782664
[===> ................] sync'ed: 23.6% (782664/1023932) K
finish: 0:04:04 speed: 3,160 (3,172) K / sec
After the synchronization is over, you can create a file system on the server where drbd is in primary state:
mkfs.xfs / dev / drbd0
For the next steps, the standard daemon launch mechanism will interfere with us. Therefore, you must run the command on both servers:
update-rc.d -f drbd remove
3. Setting heartbeat
The configuration is created in the /etc/heartbeat/ha.cf file. It is the same on both nodes, so you can create it on one and then copy it to the second.
logfacility daemon keepalive 2 deadtime 15 warntime 5 initdead 120 udpport 694 ucast eth0 10.0.0.115 ucast eth0 10.0.0.84 auto_failback on node u1 node u2 use_logd yes crm respawn
The second file is used to authenticate / etc / heartbeat / authkeys:
auth 1 1 sha1 somesecretword
It’s better not to wait for a warning from heartbeat for incorrect file attributes and change them in advance:
chmod 600 / etc / heartbeat / authkeys
After that, you can start heartbeat:
/etc/init.d/heartbeat start
After a while, the crm_mod command should show that two nodes are connected to each other:
============= Last updated: Fri Feb 10 09:33:04 2012 Stack: heartbeat Current DC: u1 (86b204d8-ee3e-47c7-ba0e-1dcbd40a20da) - partition with quorum Version: 1.0.9-unknown 2 Nodes configured, 2 expected votes 2 Resources configured. ============= Online: [u2 u1]
Next, run the crm configure edit command and enter the configuration for the cluster:
node $ id = "86b204d8-ee3e-47c7-ba0e-1dcbd40a20da" u1
node $ id = "c6e3c21f-da3e-4031-9f28-a7e33425a817" u2
primitive drbd0 ocf: linbit: drbd \
params drbd_resource = "r0" \
op start interval = "0" timeout = "240" \
op stop interval = "0" timeout = "100" \
op monitor interval = "20" role = "Slave" timeout = "20" depth = "0" \
op monitor interval = "10" role = "Master" timeout = "20" depth = "0"
primitive fs0 ocf: heartbeat: Filesystem \
params directory = "/ shared" fstype = "xfs" device = "/ dev / drbd / by-res / r0" options = "noatime, nodiratime, nobarrier, logbufs = 8" \
op start interval = "0" timeout = "60" \
op stop interval = "0" timeout = "60" \
op notify interval = "0" timeout = "60" \
op monitor interval = "20" timeout = "40" depth = "0" \
meta target-role = "Started"
primitive ip0 ocf: heartbeat: ipaddr2 \
params ip = "10.0.0.120" nic = "eth0: 0" \
op monitor interval = "5s" \
meta target-role = "Started"
primitive nfs0 ocf: itl: exportfs \
params directory = "/ shared" clientspec = "10.0.0.0/255.255.255.0" options = "rw, no_root_squash, sync, no_wdelay" fsid = "1" \
op start interval = "0" timeout = "40" \
op stop interval = "0" timeout = "60" \
op monitor interval = "30" timeout = "55" depth = "0" OCF_CHECK_LEVEL = "10" \
meta target-role = "Started"
group ha_nfs fs0 nfs0 ip0 \
meta target-role = "Started"
ms ms_drbd0 drbd0 \
meta master-max = "1" master-node-max = "1" clone-max = "2" clone-node-max = "1" notify = "true"
colocation c_nfs inf: nfs0 ms_drbd0: Master
order o_nfs inf: ms_drbd0: promote ha_nfs: start
property $ id = "cib-bootstrap-options" \
dc-version = "1.0.9-unknown" \
cluster-infrastructure = "Heartbeat" \
stonith-enabled = "false" \
expected-quorum-votes = "2" \
no-quorum-policy = "ignore" \
symmetric-cluster = "true" \
last-lrm-refresh = "1328625786"
rsc_defaults $ id = "rsc_defaults-options" \
resource-stickiness = "10000"
Exportfs settings indicate that the directory where the shared file system / shared will be mounted can be mounted by nfs to all servers on the 10.0.0.0 network.
After a while, the cluster monitor should show the status of the resources:
=============
Last updated: Fri Feb 10 09:33:04 2012
Stack: heartbeat
Current DC: u1 (86b204d8-ee3e-47c7-ba0e-1dcbd40a20da) - partition with quorum
Version: 1.0.9-unknown
2 Nodes configured, 2 expected votes
2 Resources configured.
=============
Online: [u2 u1]
Resource Group: ha_nfs
fs0 (ocf :: heartbeat: Filesystem): Started u1
nfs0 (ocf :: itl: exportfs): Started u1
ip0 (ocf :: heartbeat: ipaddr2): Started u1
Master / Slave Set: ms_drbd0
Masters: [u1]
Slaves: [u2]
4. Mount a shared file system from other servers
You can use / etc / fstab for this:
u0: / shared / var / CommuniGate / SharedDomains nfs bg, intr 0 0
5. Additional links
- When creating the configuration, the description was taken from http://library.linode.com/linux-ha/ip-failover-heartbeat-pacemaker-drbd-mysql-ubuntu-10.04
- Instructions for the treatment of split-brain drbd http://www.alsigned.ru/?p=490
- The agent is not included in Ubuntu 10.10 exportfs, so it must be downloaded separately ( https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/exportfs ) and installed in /usr/lib/ocf/resource.d/ heartbeat /
- There is no important part in this configuration - STONITH ( http://linux-ha.org/wiki/STONITH ), since there is no backup channel
- A description of the mechanism for using a shared address can be found at http://www.ultramonkey.org/3/ip_address_takeover.html