Mounting a Subdirectory of CephFS on a CentOS 7 Client
September 25, 2017 | Ceph LinuxIn this post, we describe how to mount a subdirectory of CephFS on a machine running CentOS 7, particularly how to mount a subdirectory of our Luminous Ceph filesystem on the 4-GPU workstation Hydra. For demonstration purpose, we’ll restrict Hydra to mounting only the hydra
directory of the CephFS, omitting the root directory. When you are given access to the CephFS, you’ll have your own Ceph Client username, which may be different from your UNIX username. In order to mount your own directory of the CephFS on your own machine, you should replace all occurrence of hydra
in the commands below with your own Ceph Client username.
- Add a Ceph Client User
- Restrict CephFS Client Capabilities
- Install ceph-fuse on CentOS 7 Client
- Mount a Subdirectory of CephFS
Add a Ceph Client User
So far, there is only one, default, Ceph Client user, admin, whose keyring is in the file /etc/ceph/ceph.client.admin.keyring
.
[root@pulpo-admin ~]# ceph auth get client.admin
exported keyring for client.admin
[client.admin]
key = AQxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
On one of the monitor nodes, add a new Ceph Client user hydra:
[root@pulpo-admin Pulpos]# ceph auth add client.hydra mon 'allow r' mgr 'allow r' mds 'allow rw' osd 'allow rw'
added key for client.hydra
Save user hydra’s key to a file ceph.client.hydra.keyring
, in the keyring format:
[root@pulpo-admin Pulpos]# ceph auth get-or-create client.hydra -o ceph.client.hydra.keyring
[root@pulpo-admin Pulpos]# cat ceph.client.hydra.keyring
[client.hydra]
key = AQxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
Verify user hydra’s capabilities:
[root@pulpo-admin Pulpos]# ceph auth get client.hydra
exported keyring for client.hydra
[client.hydra]
key = AQxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
caps mds = "allow rw"
caps mgr = "allow r"
caps mon = "allow r"
caps osd = "allow rw"
Make a directory hydra
on the CephFS:
[root@pulpo-dtn ~]# cd /mnt/pulpos/
[root@pulpo-dtn pulpos]# mkdir hydra
[root@pulpo-dtn pulpos]# chmod 1777 hydra
Restrict CephFS Client Capabilities
Let’s try to restrict Ceph Client hydra’s capabilities to only able to mount and work within the directory /hydra of the Ceph filesystem, following instructions in the official Ceph documentation:
[root@pulpo-admin Pulpos]# ceph fs ls
name: pulpos, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[root@pulpo-admin ~]# ceph fs authorize pulpos client.hydra /hydra rw
Error EINVAL: key for client.hydra exists but cap mds does not match
Arghhh!! As it turns out, the capacities specified by ceph auth
and those by ceph fs authorize
must exactly match!
Let’s modify Ceph Client hydra’s capabilities and try again:
[root@pulpo-admin ~]# ceph auth caps client.hydra mon 'allow r' mgr 'allow r' osd 'allow rw pool=cephfs_data' mds 'allow rw path=/hydra'
updated caps for client.hydra
[root@pulpo-admin Pulpos]# ceph auth get client.hydra
exported keyring for client.hydra
[client.hydra]
key = AQxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
caps mds = "allow rw path=/hydra"
caps mgr = "allow r"
caps mon = "allow r"
caps osd = "allow rw pool=cephfs_data"
[root@pulpo-admin ~]# ceph fs authorize pulpos client.hydra /hydra rw
[client.hydra]
key = AQxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
and it works!
Similarly, we’ve added a Ceph Client hb, for mounting the CephFS on the hummingbird cluster:
[root@pulpo-dtn ~]# cd /mnt/pulpos/
[root@pulpo-dtn pulpos]# mkdir hb
[root@pulpo-dtn pulpos]# chmod 1777 hb
[root@pulpo-admin Pulpos]# ceph auth add client.hb mon 'allow r' mgr 'allow r' osd 'allow rw pool=cephfs_data' mds 'allow rw path=/hb'
[root@pulpo-admin Pulpos]# ceph fs authorize pulpos client.hb /hb rw
[root@pulpo-admin Pulpos]# ceph auth get-or-create client.hb -o ceph.client.hb.keyring
Install ceph-fuse on CentOS 7 Client
We will not use the kernel CephFS driver, but ceph-fuse, to mount CephFS on Hydra. Although the EPEL repo provide a ceph-fuse package, it is very outdated:
[root@hydra ~]# yum info ceph-fuse
Available Packages
Name : ceph-fuse
Arch : x86_64
Epoch : 1
Version : 0.80.7
Release : 0.8.el7
Size : 1.4 M
Repo : epel/x86_64
Summary : Ceph fuse-based client
URL : http://ceph.com/
License : GPLv2
Description : FUSE based client for Ceph distributed network file system
Ceph v0.80 is codenamed Emperor, which was released in November 2013! So we won’t use ceph-fuse from the EPEL repo.
Install yum-plugin-priorities:
[root@hydra ~]# yum -y install yum-plugin-priorities
Add a Yum repository for Luminous (/etc/yum.repos.d/ceph.repo
):
[Ceph]
name=Ceph packages for $basearch
baseurl=https://download.ceph.com/rpm-luminous/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=2
[Ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-luminous/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=2
[ceph-source]
name=Ceph source packages
baseurl=https://download.ceph.com/rpm-luminous/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=2
Install ceph-fuse:
[root@hydra ~]# yum install ceph-fuse
[root@hydra ~]# ceph-fuse --version
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)
Mount a Subdirectory of CephFS
I) Create the directory /etc/ceph
on Hydra:
[root@hydra ~]# mkdir /etc/ceph
II) Copy ceph.conf
& ceph.client.hydra.keyring
to /etc/ceph on Hydra.
III) Change the permission of /etc/ceph/ceph.client.hydra.keyring
so that only root can read and write it:
[root@hydra ~]# chmod 600 /etc/ceph/ceph.client.hydra.keyring
IV) Create the mountpoint, e.g., /mnt/pulpos
:
[root@hydra ~]# mkdir /mny/pulpos
Now we are ready to mount the subdirectory /hydra
of the CephFS, at the mountpoint /mnt/pulpos
on Hydra. There are 3 options:
1) Manually run ceph-fuse
command:
[root@hydra ~]# ceph-fuse -n client.hydra -m 128.114.86.4:6789 -r /hydra /mnt/pulpos
or more redundantly:
[root@hydra ~]# ceph-fuse -n client.hydra -m pulpo-mon01.ucsc.edu:6789,pulpo-mds01.ucsc.edu:6789,pulpo-admin.ucsc.edu:6789 -r /hydra /mnt/pulpos
2) To mount the subdirectory /hydra
of the CephFS automatically on startup, we can add the following to /etc/fstab
(see http://docs.ceph.com/docs/luminous/cephfs/fstab/#fuse):
none /mnt/pulpos fuse.ceph ceph.id=hydra,ceph.client_mountpoint=/hydra,defaults,_netdev 0 0
the we can manually mount it with:
[root@hydra ~]# mount /mnt/pulpos/
3) Another option to automate mounting of the subdirectory /hydra
of the CephFS is to use systemd. To take this route, we first need to modify the unit file /usr/lib/systemd/system/ceph-fuse@.service
:
[Unit]
Description=Ceph FUSE client
After=network-online.target local-fs.target time-sync.target
Wants=network-online.target local-fs.target time-sync.target
Conflicts=umount.target
PartOf=ceph-fuse.target
[Service]
EnvironmentFile=-/etc/sysconfig/ceph
Environment=CLUSTER=ceph
ExecStart=/usr/bin/ceph-fuse -f --cluster ${CLUSTER} -n client.hydra -r /hydra %I
TasksMax=infinity
Restart=on-failure
StartLimitInterval=30min
StartLimitBurst=3
[Install]
WantedBy=ceph-fuse.target
Note that we’ve add the flags -n client.hydra -r /hydra
to ceph-fuse in the ExecStart line of the unit file.
Reload systemd manager configuration (because we’ve made changes to the unit file for ceph-fuse@.service):
[root@hydra ~]# systemctl daemon-reload
Start the ceph-fuse service to mount the subdirectory /hydra
of the CephFS, at the mountpoint /mnt/pulpos
:
[root@hydra ~]# systemctl start ceph-fuse@/mnt/pulpos.service
To create a persistent mount point:
[root@hydra ~]# systemctl enable ceph-fuse.target
[root@hydra ~]# systemctl enable ceph-fuse@-mnt-pulpos
NOTE here the command must be systemctl enable ceph-fuse@-mnt-pulpos
. If we run systemctl enable ceph-fuse@/mnt/pulpos
instead, we’ll get an error “Failed to execute operation: Unit name pulpos is not valid.” However, when starting the service, we can run either systemctl start ceph-fuse@/mnt/pulpos
or systemctl start ceph-fuse@-mnt-pulpos
!
Lastly, we note the same bug in current version of Luminous (v12.2.1): when CephFS is mounted using ceph-fuse, the mount point doesn’t show up in the output of df
; and although we can list the mount point specifically with df -h /mnt/pulpos
, the size of the filesystem is reported as 0!
[root@hydra ~]# df -h /mnt/pulpos
Filesystem Size Used Avail Use% Mounted on
ceph-fuse 0 0 0 - /mnt/pulpos