If you’ve been following Project Titanicarus, you’ll know that I’ve had a reasonably serious love/hate relationship with clustering filesystems. I’ve been using DRBD and OCFS2 in an active/active configuration for the last 6 months or so. The experience has been ok, but I’m only saying that because of the other horrendous options I’ve tried.
Long & short of it, DRBD and OCFS2 suck. They are painful, I’ve had to write scripts that help them auto recover and even after covering about 10 different failure scenarios I still have downtime on a weekly basis when things don’t go well.
CEPH is part of the Openstack project, it provides a scalable, distributed multi-node striped filesystem that can be mounted as a block level device or using CEPH-FS, CEPH’s own clustering filesystem.
Installing stuff.
For this project I’m installing CEPH on a pair of servers.
Each server has 3 disks attached:
- /dev/sda 20Gb System Disk
- /dev/sdb 25Gb Journal Disk
- /dev/sdc 500Gb Storage Disk
I’m setting both servers up to be OSD, monitor and metadata servers. I followed this guide to get things rolling, if you have any issues with my instructions, try using it.
Install instructions are as follows:
# create partions on each server
parted /dev/sdc (parted) mklabel gpt (parted) mkpart primary xfs 0 100% (parted) quit
parted /dev/sdc (parted) mklabel gpt (parted) mkpart primary 0 100% (parted) quit
mkfs.xfs /dev/sdb1 mkfs.xfs /dev/sdc1
# Install ceph-deploy tools on your admin node echo deb http://ceph.com/debian/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
apt-get update apt-get install ceph-deploy # create a user to manage ceph from on your admin node
useradd -d /home/cluster-admin -m cluster-admin -s /bin/bash passwd cluster-admin su cluster-admin # create a ceph user on each node
useradd -d /home/ceph -m ceph -s /bin/bash
passwd ceph
echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
chmod 0440 /etc/sudoers.d/ceph # add hosts file entries for each of your hosts
cat /etc/hosts 127.0.0.1 localhost 10.0.0.50 filer01 10.0.0.51 filer02 # Create a public key on your admin node (as cluster-admin) and push out to all nodes
ssh-keygen
ssh-copy-id ceph@filer01
ssh-copy-id ceph@filer02
# Setup an SSH access configuration by editing the .ssh/config file
Host filer01
Hostname filer01
User ceph
Host filer02
Hostname filer02
User ceph # create a folder to work from in the cluster-admin home dir
mkdir ceph-cluster cd ceph-cluster # create the new monitor servers
ceph-deploy new filer01 filer02 # Tell ceph its ok to only have 2 filers
echo "[default]" >> ceph.conf
echo'osd pool default size = 2'>> ceph.conf
#install ceph on all nodes
ceph-deploy install --no-adjust-repos filer01 filer02
# create monitor and gather keys
ceph-deploy mon create-initial
# deploy to disks
ceph-deploy disk zap --fs-type xfs filer01:/dev/sdc1
ceph-deploy disk zap --fs-type xfs filer02:/dev/sdc1
ceph-deploy osd prepare filer01:sdc1:sdb1 filer02:sdc1:sdb1
ceph-deploy osd activate filer01:sdc1:sdb1 filer02:sdc1:sdb1
# copy keys and configs to all hosts
ceph-deploy admin filer01 filer02
# set permissions on keyring
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
# setup the metadata servers required for cephfs
ceph-deploy mds create filer01 filer02
And you’re done!
To check the health of the ceph cluster run the following commands:
ceph health ceph check
If either of those show errors, something not right and you need to check it out.
Mounting your datastore
To mount your datastore, open up /etc/ceph/ceph.client.admin.keyring and grab the key out of it.
Create a mount point for the new storage and mount it.
mkdir /data/ceph
mount -t ceph
filer01:6789,filer02:6789:/ /data/ceph -o name=admin,secret=XQXLXIX1XdXXRXXXXXXtqUmI2XXXXXX==
You should now see your datastore mounted.
Conclusion
So far this has been pretty easy. I’m going to try and run a few websites and some email users on ceph this week. I will let you know how it fares.