In this post, we take a look at security on the PRP Kubernetes cluster. The Kubernetes cluster is experimental/alpha in its current incarnation, so security can be understandably quite lax.

• CILOGON
• Docker Security
• Kubernetes Security
• Discussion

CILOGON

All Kubernetes clusters have two categories of users: service accounts managed by Kubernetes, and normal users. The latter are assumed to be managed by an outside, independent service. For the PRP Kubernetes cluster, the outside service is CILogon. Currently there is no restriction: anyone with a valid CILogon credential can login at the PRP Kubernetes gateway. Then he/she can download a Kubeconfig file and launch containers on the cluster. Since CILogon bridges the identity credentials generated by over 200 universities and organizations (through the InCommon Federation) as well as Google identity credentials, virtually anyone in the world can freely use the cluster, including the four GPU nodes! Given the stratospherically (some may say bubbly) high prices for cryptocurrencies, I am constantly amazed that no one has used the GPU nodes to mine bitcoins and the likes for free yet!

Docker Security

To put it mildly, security is not Docker’s forte. root (id = 0) is the default user within a container, even though the container is launched by a normal user (a normal user must belong to the docker group in order to be able to run docker commands). For example, on the 4-GPU workstation Hydra:

[dong@hydra ~]$ id
uid=1000(dong) gid=1000(dong) groups=1000(dong),993(docker)
[dong@hydra ~]$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Nov 29 12:01 /var/run/docker.sock

[dong@hydra ~]$ docker run -ti --rm alpine sh
/ # id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
/ # exit

If we mount a host volume / directory on the container, we’ll have unfettered, read and write, access to the files in the volume:

[dong@hydra ~]$ docker run -ti --rm -v /etc:/etc2 alpine sh
/ # df -h /etc2
Filesystem                Size      Used Available Use% Mounted on
/dev/md125              101.6G     25.5G     76.1G  25% /etc2

Furthermore, if we executes docker run with the --privileged flag, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host! For example, we can mount any host block device on the container, and do whatever tickles our fancy.

[dong@hydra ~]$ docker run -ti --rm --privileged alpine sh

We can use the -u (or --user) to run the container as a non-root user. When passing a numeric ID, the user does not have to exist in the container, not in the host:

[dong@hydra ~]$ docker run -ti --rm --user=2000:2000 alpine sh
/ $ id
uid=2000 gid=2000

On the host, we can see that the sh process is running as uid 2000:

[root@hydra ~]# ps -u 2000
  PID TTY          TIME CMD
  496 pts/0    00:00:00 sh

This scheme, however, won’t mitigate Docker’s security risks. It will at best protects the host from a malicious Docker image; but it won’t protect the host from a troublesome user!

Kubernetes Security

First switch to normal user shaw@ucsc.edu:

$ kubectl config use-context shaw
Switched to context "shaw".

Next create a Kubernetes manifest alpine-volume.yaml, which will mount the host’s /etc directory at /etc2 on the alpine container:

apiVersion: v1
kind: Pod
metadata:
  name: alpine
spec:
  containers:
  - name: alpine
    image: alpine
    imagePullPolicy: Always
    args: ["sleep", "36500000"]
    volumeMounts:
    - name: etc
      mountPath: /etc2
  restartPolicy: Never
  volumes:
    - name: etc
      hostPath:
        path: /etc

Create a pod object from the YAML manifest:

$ kubectl create -f alpine-volume.yaml
pod "alpine" created

Check whether the pod is running:

$ kubectl get pod alpine
NAME      READY     STATUS    RESTARTS   AGE
alpine    1/1       Running   0          1m

Start a shell in the running pod:

$ kubectl exec -it alpine -- sh
/ # df -h /etc2
Filesystem                Size      Used Available Use% Mounted on
/dev/md126              222.8G     16.1G    206.8G   7% /etc2
/ # cat /etc2/hostname
k8s-nvme-01.sdsc.optiputer.net
/ # exit

We can see that the host’s /etc is indeed mounted at /etc2 on the container; and we have unfettered access to the files therein.

Clean up by deleting the pod:

$ kubectl delete -f alpine-volume.yaml
pod "alpine" deleted

For the last experiment, create a Kubernetes manifest alpine-privileged.yaml, which will create a privileged container:

apiVersion: v1
kind: Pod
metadata:
  name: alpine
spec:
  containers:
  - name: alpine
    image: alpine
    imagePullPolicy: Always
    args: ["sleep", "36500000"]
    securityContext:
      privileged: true
  restartPolicy: Never

Create a pod object from the YAML manifest:

$ kubectl create -f alpine-privileged.yaml
pod "alpine" created

Check whether the pod is running:

$ kubectl get pod alpine
NAME      READY     STATUS    RESTARTS   AGE
alpine    1/1       Running   0          14s

Start a shell in the running pod:

$ kubectl exec -it alpine -- sh
/ # ls /dev
/ # exit

We find that indeed the container has full access to all devices on the host!

Clean up by deleting the pod:

$ kubectl delete -f alpine-privileged.yaml
pod "alpine" deleted

Discussion

These security risks are pretty serious! Perhaps we should look into Security Context and Pod Security Policy to mitigate these risks?