High Availability NFS with DRBD and HeartBeat DRBD allows you to create a mirror of two block devices that are located at two different sites across an IP network on Linux platform. DRBD replicates data on the primary device to the secondary device in a way that ensures that both copies of the data remain identical. DRBD works on top of block devices, i.e., hard disk partitions or LVM’s logical volumes. It mirrors each data block that it is written to disk to the peer node.
Mostly DRBD allows you to use any block device like this.
partition or complete hard disk
software RAID
Logical Volume Manager (LVM)
Enterprise Volume Management System (EVMS) Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allows clients to know about the presence (or disappearance!) of peer processes on other machines and to easily exchange messages with them.
Use the following tutorial to configure High Availability nfs storage with DRBD and HeartBeat for your centos 6 server.
Redundant NFS with DRBD and HeartBeat Requirements
– Install NFS
Install NFS server if you were not installed.
#yum install nfs-utils nfs-utils-lib
– Two drives with same size (preferably)
– Networking between storage1 and storage2 servers.
– Working DNS resolution
ex:
/etc/hosts
10.1.1.2 storage1.clouddb.net
10.1.1.3 storage1.clouddb.net
– install NTP synchronized times on both server.
Install NTP command line if you were not installed.
#yum install ntp
– Set selinux Permissive mode
– Enable port 7788 on iptables.
We have used Virtual (VIP) ipaddress to configure heartbeat and prepare the following server details and partition before proceed.
storage1.clouddb.net
IP : 10.1.1.2
storage2.clouddb.net
IP : 10.1.1.3
Virtual (VIP) ipaddress : 10.1.1.4
Same partition on both server
/dev/sdb1
Update /data folder on NFS share /etc/exports file.
Insert drbd module manually on both machines or reboot:
storage1/storage2
#modprobe drbd
Next we need to create a new DRBD resource file by creating /etc/drbd.d/loadbalance.res. Make sure to use the correct IP address and devices for your server. As resource name you can use whatever you like.
#vi /etc/drbd.d/loadbalance.res
on storage1.cloudkb.net {
device /dev/drbd0;
disk /dev/xvdc1;
address 10.1.1.2:7788;
meta-disk internal;
}
on storage2.cloudkb.net {
device /dev/drbd0;
disk /dev/xvdb1;
address 10.1.1.3:7788;
meta-disk internal;
} }
Copy /etc/drbd.d/loadbalance.res on storage2 server.
scp /etc/drbd.d/loadbalance.res storage2.clouddb.net:/etc/drbd.d//etc/drbd.d/loadbalance.res
Initialize the DRBD meta data storage on both machines:
storage1/storage2
# drbdadm create-md loadbalance
Check below error section if you have any errors.
Start the drdb on both servers:
storage1/storage2
# service drbd start
On the primary ( Storage1 ) node run drbdadm command:
#drbdadm primary loadbalance
On the secondary ( storage2 )node run drbdadm command:
#drbdadm secondary loadbalance
You can check the initial synchronization progress by checking /proc/drbd.
example,
[root@storage1 ~]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r—n-
ns:788174376 nr:72 dw:11979700 dr:776199088 al:3180 bm:47383 lo:0 pe:3 ua:2 ap:0 ep:1 wo:f oos:22787584
[==================>.] sync’ed: 97.2% (22252/780380)M
finish: 0:46:12 speed: 8,204 (8,192) K/sec
When the initial sync is finished, the output should look like this:
[root@storage1 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2013-09-27 16:00:43
0: cs:Connected ro:Secondary/Primary ds:Diskless/UpToDate C r—–
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Almost done, You should create filesystem on Distributed Replicated Block Device device and mount it.
# mkfs.ext4 /dev/drbd0
# mkdir /data
# mount /dev/drbd0 /data
[root@storage1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
29G 5.0G 22G 19% /
tmpfs 245M 0 245M 0% /dev/shm
/dev/sda1 485M 53M 407M 12% /boot
/dev/drbd0 5.0G 139M 4.6G 3% /data
You don’t need to mount the disk from storage1 machines. All your data write on /data folder will be synced to storage2. To check this data write, unmount /data folder on storage1 and mount to storage2
## RHEL/CentOS 6 64-Bit ##
# wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
# rpm -ivh epel-release-6-8.noarch.rpm
Once the heartbeat installation completed, update the following configuraiton on both server.
storage2.clouddb.net IPaddr::10.1.1.4/24 drbddisk::loadbalance Filesystem::/dev/drbd0::/data::ext4 nfs
Thats all!!!.. You have completed the setup. Try to restart heartbeat service.
#/etc/init.d/heartbeat start
You can monitor the log from /var/log/messages.
You should check whether storage server have mounted /data folder and you can use 10.1.1.4 ip address to mount NFS on your another servers.
Note : check 10.1.1.4 ping request whether its working.
For example:
You can mount /data folder on Host machine.
If you have any issue with drdb synchronize, On the primary only run the following command to initialize the synchronization between the two nodes.
drbdadm primary –force loadbalance
Checking the status reveals:
cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r—–
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:1953460984
The reason for this is because neither server can tell if it has the correct data, so we need to invalidate one of them, so the other one thinks it is up-to-date. On the secondary server, simple use this command.
Found ext3 filesystem
937692472 kB data area apparently used
937663816 kB left usable by current configuration
Device size would be truncated, which
would corrupt data and result in
‘access beyond end of device’ errors.
You need to either
* use external meta data (recommended)
* shrink that filesystem first
* zero out the device (destroy the filesystem)
Operation refused.
Command ‘drbdmeta 0 v08 /dev/sdb1 internal create-md’ terminated with exit code 40
drbdadm create-md tld6drbd: exited with code 40
Fix:
dd if=/dev/zero of=/dev/sdb1 bs=1M count=128
DRBD & mkfs: Wrong medium type while trying to determine filesystem size
[root@primary05 ~]# mkfs -t ext4 /dev/drbd0
mke2fs 1.41.12 (17-May-2010)
mkfs.ext4: Wrong medium type while trying to determine filesystem size
Cause:
The state of your device is kept in /proc/drbd:
[root@primary05 ~]# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r—-
ns:0 nr:18104264 dw:18100168 dr:0 al:0 bm:0 lo:1024 pe:0 ua:1025 ap:0 ep:2 wo:b oos:0
The part ro:Secondary/Primary tells you what both nodes are set to.
Fix:
Switch the DRBD block device to Primary using drbdadm:
# drbdadm primary loadbalance
If you have any problem to sync between two servers, then check the /var/log/message.
