Systemy plików w klastrach
Debian Wheezy DRBD primary/primary + Corosync + CLVM + OCFS2
Posted: 2013/06/10 | Author: ;-) | Filed under: linux | Tags: cluster lvm, clvm, corosync, drbd dual master, drbd dual primary, drbd master master, drbd primary primary, ocfs, ocfs2, pacemaker |Leave a comment »
How to install :
- DRBD in dual primary mode,
- Corosync + Pacemaker,
- CLVN,
- OCFS2
on Debian Wheezy.I am going to use two servers xbox-1 (192.168.50.11) and xbox-2 (192.168.50.12) with Debian 7 (Wheezy). Each server will have the following partition layout :
- /dev/sda1 and /dev/sda2 for boot and root,
- /dev/sda3 will be replicated as /dev/drbd0xboxIf not specified otherwise all configurations should be made on both nodes.1. DRBD1.1 Installapt-get install drbd8-utils
1.2 Configure/etc/drbd.d/global_common.confglobal {
usage-count yes;
}
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
}
startup {
wfc-timeout 0;
degr-wfc-timeout 120;
become-primary-on both;
}
disk {
on-io-error detach;
}
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
rr-conflict call-pri-lost;
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 0;
}
syncer {
rate 100M;
al-extents 3389;
}
}
/etc/drbd.d/sda3.resresource sda3 {
device /dev/drbd0;
disk /dev/sda3;
meta-disk internal;
on xbox-1 {
address 192.168.50.11:7788;
}
on xbox-2 {
address 192.168.50.12:7788;
}
}
Prevent drbd from starting automatically (it will be started by corosync)update-rc.d drbd disable
1.3 Prepare drbd resourcesdrbdadm create-md sda3
1.4 Start drbd/etc/init.d/drbd start
On xbox-1 :drbdadm -- --overwrite-data-of-peer primary sda3
At this point if you run drbd-overview you should see the nodes connected and synchronization started.2. Corosync and Pacemaker2.1 Installapt-get install corosync pacemaker openais
2.1 Configure corosync (in my case I use unicast communication)./etc/corosync/corosync.conftotem {
version: 2
token: 3000
token_retransmits_before_loss_const: 10
join: 60
consensus: 3600
vsftype: none
max_messages: 20
clear_node_high_bit: yes
secauth: off
threads: 0
rrp_mode: active interface {
ringnumber: 0
member {
memberaddr: 192.168.50.11
}
member {
memberaddr: 192.168.50.12
}
# on xbox-1
bindnetaddr: 192.168.50.11
# on xbox-2
#bindnetaddr: 192.168.50.12
mcastport: 5405
} transport: udpu
}amf {
mode: disabled
}service {
ver: 0
name: pacemaker
}aisexec {
user: root
group: root
}logging {
fileline: off
to_stderr: no
to_syslog: yes
syslog_facility: daemon
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
tags: enter|leave|trace1|trace2|trace3|trace4|trace6
}
}
2.2 Start corosync/etc/init.d/corosync start
At this point if you run crm_mon (or crm status) you should see, after a while, both nodes online2.3 Some basic 2-nodes cluster settings (all crm commands have to be run only on one node)crm configureproperty $id="cib-bootstrap-options" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
op_defaults $id="op-options" \
interval="0"
3. Add drbd to clustercrm configure
primitive p-drbd ocf:linbit:drbd \
params drbd_resource="sda3" \
op monitor interval="50" role="Master" timeout="30" \
op monitor interval="60" role="Slave" timeout="30" \
op start interval="0" timeout="240" \
op stop interval="0" timeout="100"
ms ms-drbd p-drbd \
meta master-max="2" clone-max="2" notify="true" interleave="true"
4. DLM (Distributed Lock Manager)4.1 Installapt-get install dlm-pcmk
4.2 Configurecrm configure
primitive p-dlm ocf:pacemaker:controld \
op monitor interval="120" timeout="30" \
op start interval="0" timeout="90" \
op stop interval="0" timeout="100"
5. CLVM5.1 Installapt-get install clvm
Debian Wheezy is missing the clvm ocf resource agent (/usr/lib/ocf/resource.d/lvm2/clvmd), so I took it from the ubuntu clvm package.5.2 ConfigureDisable automatic startupupdate-rc.d clvm disable
In /etc/lvm/lvm.conf changelocking_type = 3
Cluster configurationcrm configureprimitive p-clvm ocf:lvm2:clvmd \
params daemon_timeout="30" \
op monitor interval="60" timeout="30" \
op start interval="0" timeout="90" \
op stop interval="0" timeout="100"
6. OCFS26.1 Installapt-get install ocfs2-tools ocfs2-tools-pacemaker
6.2 Configcrm configure
primitive p-o2cb ocf:pacemaker:o2cb \
op monitor interval="120" timeout="30" \
op start interval="0" timeout="90" \
op stop interval="0" timeout="100"
Disable automatic startupupdate-rc.d ocfs2 disable
update-rc.d o2cb disable
7. Put everything togetherAdd the rest of cluster conf which will make everything work togethercrm configure
group g-lock p-dlm p-clvm p-o2cb
clone c-lock g-lock \
meta globally-unique="false" interleave="true"
colocation col-drbd-lock inf: c-lock ms-drbd:Master
order ord-drbd-lock inf: ms-drbd:promote c-lock
At this point if you run crm status you should see all cluster resources up and runningOnline: [ xbox-1 xbox-2 ]Master/Slave Set: ms-drbd [p-drbd]
Masters: [ xbox-2 xbox-1 ]
Clone Set: c-lock [g-lock]
Started: [ xbox-1 xbox-2 ]
and you should be able to create lvm volumes on one node and access them on the other node.8. Create LVM volumes and shared file system (all commands have to be run only on one node)8.1 LVM stuffpvcreate /dev/drbd0
vgcreate vg0 /dev/drbd0
lvcreate -n sharelv -L 10G vg0
If you run vgdisplay -v on the other node you should see vg0 and sharelv.8.2 Ocfs2mkfs.ocfs2 /dev/vg0/sharelv
mkdir /root/sharedata
8.3 Cluster configcrm configureprimitive p-vg0 ocf:heartbeat:LVM \
params volgrpname="vg0" \
op monitor interval="60s" timeout="40" \
op start interval="0" timeout="40" \
op stop interval="0" timeout="40"
primitive p-sharefs ocf:heartbeat:Filesystem \
params device="/dev/vg0/sharelv" directory="/sharedata" fstype="ocfs2" \
op monitor interval="60s" timeout="60s" \
op start interval="0" timeout="90s" \
op stop interval="0" timeout="90s"
group g-sharedata p-vg0 p-sharefs
clone c-sharedata g-sharedata \
meta globally-unique="false" interleave="true"
colocation col-lock-sharedata inf: c-sharedata c-lock
order ord-lock-sharedata inf: c-lock c-sharedata
At this point if you run crm status you should see all cluster resources up and runningOnline: [ xbox-1 xbox-2 ]Master/Slave Set: ms-drbd [p-drbd]
Masters: [ xbox-2 xbox-1 ]
Clone Set: c-lock [g-lock]
Started: [ xbox-1 xbox-2 ]
Clone Set: c-sharedata [g-sharedata]
Started: [ xbox-1 xbox-2 ]
and /dev/vg0/sharelv should be mounted r/w on /shardata on both nodes.