Project Titanicarus: Part 10 – Adventures with CEPH

If you’ve been following Project Titanicarus, you’ll know that I’ve had a reasonably serious love/hate relationship with clustering filesystems. I’ve been using DRBD and OCFS2 in an active/active configuration for the last 6 months or so. The experience has been ok, but I’m only saying that because of the other horrendous options I’ve tried.

Long & short of it, DRBD and OCFS2 suck. They are painful, I’ve had to write scripts that help them auto recover and even after covering about 10 different failure scenarios I still have downtime on a weekly basis when things don’t go well.

CEPH is part of the Openstack project, it provides a scalable, distributed multi-node striped filesystem that can be mounted as a block level device or using CEPH-FS, CEPH’s own clustering filesystem.CEPH Logo

Installing stuff.

For this project I’m installing CEPH on a pair of servers.

Each server has 3 disks attached:

  • /dev/sda 20Gb System Disk
  • /dev/sdb 25Gb Journal Disk
  • /dev/sdc 500Gb Storage Disk

I’m setting both servers up to be OSD, monitor and metadata servers. I followed this guide to get things rolling, if you have any issues with my instructions, try using it.

Install instructions are as follows:

# create partions on each server
parted /dev/sdc
(parted) mklabel gpt
(parted) mkpart primary xfs 0 ­100%
(parted) quit
parted /dev/sdc
(parted) mklabel gpt
(parted) mkpart primary 0 100%
(parted) quit
mkfs.xfs /dev/sdb1
mkfs.xfs /dev/sdc1
# Install ceph-deploy tools on your admin node
echo deb $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
wget -q -O- ';a=blob_plain;f=keys/release.asc' | sudo apt-key add -
apt-get update
apt-get install ceph-deploy

# create a user to manage ceph from on your admin node
useradd -d /home/cluster-admin -m cluster-admin -s /bin/bash
passwd cluster-admin
su cluster-admin

# create a ceph user on each node
useradd -d /home/ceph -m ceph -s /bin/bash
passwd ceph
echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
chmod 0440 /etc/sudoers.d/ceph

# add hosts file entries for each of your hosts
cat /etc/hosts       localhost       filer01       filer02

# Create a public key on your admin node (as cluster-admin) and push out to all nodes
ssh-copy-id ceph@filer01
ssh-copy-id ceph@filer02
# Setup an SSH access configuration by editing the .ssh/config file
Host filer01
   Hostname filer01
   User ceph
Host filer02
   Hostname filer02
   User ceph

# create a folder to work from in the cluster-admin home dir
mkdir ceph-cluster
cd ceph-cluster

# create the new monitor servers
ceph-deploy new filer01 filer02

# Tell ceph its ok to only have 2 filers
echo "[default]" >> ceph.conf
echo'osd pool default size = 2'>> ceph.conf
#install ceph on all nodes
ceph-deploy install --no-adjust-repos filer01 filer02
# create monitor and gather keys
ceph-deploy mon create-initial
# deploy to disks
ceph-deploy disk zap --fs-type xfs filer01:/dev/sdc1
ceph-deploy disk zap --fs-type xfs filer02:/dev/sdc1
ceph-deploy osd prepare filer01:sdc1:sdb1 filer02:sdc1:sdb1
ceph-deploy osd activate filer01:sdc1:sdb1 filer02:sdc1:sdb1
# copy keys and configs to all hosts
ceph-deploy admin filer01 filer02
# set permissions on keyring
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
# setup the metadata servers required for cephfs
ceph-deploy mds create filer01 filer02

And you’re done!

To check the health of the ceph cluster run the following commands:

ceph health
ceph check

If either of those show errors, something not right and you need to check it out.

Mounting your datastore

To mount your datastore, open up /etc/ceph/ceph.client.admin.keyring and grab the key out of it.

Create a mount point for the new storage and mount it.

mkdir /data/ceph
mount -t ceph
filer01:6789,filer02:6789:/ /data/ceph -o name=admin,secret=XQXLXIX1XdXXRXXXXXXtqUmI2XXXXXX==

You should now see your datastore mounted.


So far this has been pretty easy. I’m going to try and run a few websites and some email users on ceph this week. I will let you know how it fares.

Last updated by at .