DRBD 8.3.0

DRBD 8.3.0 installation & prerequisites

For drbd testing, two working Ubuntu Server Jaunty installations are needed. This can be done with ubuntu-vm-builder or tests can be done on real hardware. Both servers need to be able to reach each other over the network.

Each server needs one additional partition (additional to the one for the system). The size of partition isn't relevant for testing, but both partitions on servers need to be of the same size.

For the purpose of this tutorial, I'll name them drbd-1 and drbd-2. Servers need to be able to resolve each others hostnames, so you should either have DNS or enter hostnames to /etc/hosts manually. Since drbd can start before dhcp client gets an IP, you should set up both servers with static IPs.

/etc/hosts:

127.0.0.1 localhost
192.168.0.1 drbd-1
192.168.0.2 drbd-2

From this point you should do everything as root (sudo -i).

Next, install drbd8-utils package.

Configuration

drbd has one single configuration file - /etc/drbd.conf. This file should be identical on both servers. This file should look like this:

global { usage-count no; }
common { syncer { rate 100M; } }
resource r0 {
        protocol C;
        startup {
                wfc-timeout  15;
                degr-wfc-timeout 60;
        }
        net {
                cram-hmac-alg sha1;
                shared-secret "test";
                allow-two-primaries;
        }
        on drbd-1 {
                device /dev/drbd0;
                disk /dev/sdb1;
                address 192.168.0.1:7788;
                meta-disk internal;
        }
        on drbd-2 {
                device /dev/drbd0;
                disk /dev/sdb1;
                address 192.168.0.2:7788;
                meta-disk internal;
        }
}

Once you've created this file on both servers, on both of them run command:

drbdadm create-md r0

This will create r0 resource on both servers. Then, choose one server to be your primary server (drbd-1), and mark it:

drbdadm -- --overwrite-data-of-peer primary all

As soon as you run this command, the other server (drbd-2) will start syncing data. You can check the progress with:

cat /proc/drbd

You should get something like this:

version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by ivoks@ubuntu, 2009-01-17 07:49:56
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---
    ns:76524 nr:0 dw:0 dr:84352 al:0 bm:4 lo:1 pe:11 ua:245 ap:0 ep:1 wo:b oos:429772
       [==>.................] sync'ed: 16.2% (429772/505964)K
       finish: 0:00:22 speed: 19,048 (19,048) K/sec

Once syncing is done, this will change to:

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by ivoks@ubuntu, 2009-01-17 07:49:56
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r---
    ns:505964 nr:0 dw:0 dr:505964 al:0 bm:31 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

Tests

A: ext3 in primary/secondary

Create ext3 fileystem on primary server (drbd-1):

mkfs.ext3 /dev/drbd0

Mount special device /dev/drbd0 to /mnt/drbd and copy a file to it:

mkdir /mnt/drbd
mount -t ext3 /dev/drbd0 /mnt/drbd
cp /etc/hosts /mnt/drbd

Unmount the partition and degrade primary to secondary (drbd-1):

umount /mnt/drbd
drbdadm secondary r0

On the other server (drbd-2), promote it to primary, mount drbd device and check copied filed:

drbdadm primary r0
mkdir /mnt/drbd
mount -t ext3 /dev/drbd0 /mnt/drbd
ls -d /mnt/drbd/*

NOTE: At this point, primary is no longer drbd-1, but drbd-2, and secondary is drbd-1.

B: secondary off

NOTE: Run test A first.

Shutdown secondary server (drbd-1). Check /proc/drbd:

cat /proc/drbd

You should see that the other server is unavailable:

version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by ivoks@ubuntu, 2009-01-17 07:49:56
 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r---
    ns:11 nr:530744 dw:530767 dr:561 al:2 bm:31 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:24

Create couple of files on primary server (drbd-2):

cp /etc/hosts.* /etc/group /mnt/

Now, power up secondary server (drbd-1). As soon as it boots, it will start syncing data. If you are testing will small partitions, sync will be faster. On primary server (drbd-2), /proc/drbd should look like this:

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by ivoks@ubuntu, 2009-01-17 07:49:56
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r---
    ns:63 nr:530744 dw:530788 dr:614 al:2 bm:33 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

C: primary off

NOTE: Run test A first. Running test B isn't needed for this one, but it doesn't hurt.

Shutdown primary server (drbd-2) with device /dev/drbd0 mounted. /proc/drbd on secondary (drbd-1), after few seconds, should look like this:

version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by ivoks@ubuntu, 2009-01-17 07:49:56
 0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r---
    ns:0 nr:53 dw:53 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

Make it primary:

drbdadm primary r0

NOTE: From this point, primary is again drbd-1, and secondary is again drbd-2.

Now, you should be able to mount /dev/drbd0 on primary (drbd-1):

mount -t ext3 /dev/drbd0 /mnt/drbd

Once secondary (drbd-2) boots up again, it won't connect to primary, since it was primary before the power failure and they can't decide which will be primary again. This is called split-brain situation. On secondary (drbd-2) run this commands:

drbdadm secondary r0
drbdadm -- --discard-my-data connect r0

And on the primary (drbd-1) run this command:

drbdadm connect r0

This will bring them back in sync. cat /proc/drbd:

version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by ivoks@ubuntu, 2009-01-17 07:49:56
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r---
    ns:8192 nr:0 dw:62 dr:8598 al:2 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

D: primary-primary with cluster-suite

NOTE: Previous test are optional. This test requires installation of additional programs.

Install redhat-cluster-suite package. After installation, create on both servers file /etc/cluster/cluster.conf:

<?xml version="1.0" ?>
<cluster config_version="2" name="drbd">
       <fence_daemon post_fail_delay="0" post_join_delay="3"/>
       <clusternodes>
               <clusternode name="drbd-1" nodeid="1" votes="1">
                       <fence/>
               </clusternode>
               <clusternode name="drbd-2" nodeid="2" votes="1">
                       <fence/>
               </clusternode>
       </clusternodes>
       <cman expected_votes="1" two_node="1"/>
       <fencedevices>
               <fencedevice agent="fence_xvm" name="KVM"/>
       </fencedevices>
       <rm>
               <failoverdomains>
                       <failoverdomain name="drbd" ordered="0" restricted="0">
                               <failoverdomainnode name="drbd-1" priority="1"/>
                               <failoverdomainnode name="drbd-2" priority="1"/>
                       </failoverdomain>
               </failoverdomains>
               <resources/>
       </rm>
</cluster>

Reboot both servers. After they come up, check if both servers are in cluster. On both, run:

cman_tool status | grep 'Cluster Member'

Both should return Cluster Member: yes. Now, set up both servers as primary. On both, run:

drbdadm primary r0

On one server, create GFS2 filesystem:

mkfs.gfs2 -j 2 -p lock_dlm -t drbd:drbd /dev/drbd0

mkfs will report existing filesystem, so in that case, just answer y on asked question.

After filesystem is created, mount the filesystem on both servers:

mkdir /mnt/drbd (create directory if you skipped test A)
mount /dev/drbd0 /mnt/drbd

Now write a file on one server and check if that file exists on the other server. If that's true, congratulations, you've successfully tested low-cost-easy-to-setup alternative for iSCSI.

Test results