k3s on Raspberry Pi: Persistent Storage using hostPath

16 Dec 2021 18:20 k3s raspberry-pi

We need to give our Docker registry some persistent storage. Currently, if we restart it, it loses its stored data.


We can specify hostPath directly in the container specification. That allows us to mount a file or directory from the host.

To do that, edit the docker-registry.yml file from before.

It should look like this:

apiVersion: apps/v1
kind: Deployment
  name: docker-registry
  namespace: docker-registry
    app: docker-registry
  replicas: 1
      app: docker-registry
        app: docker-registry
        name: docker-registry
      - name: docker-registry-vol
          path: /tmp/docker-registry
      - name: registry
        image: registry:2
        - containerPort: 5000
        - name: docker-registry-vol
          mountPath: /var/lib/registry

We’ve added the volumes and volumeMounts sections. These define a volume (using hostPath, so that it’s stored on the node’s filesystem), and a volume mount (which mounts that volume inside the container).

Apply it:

$ kubectl apply -f docker-registry.yml
deployment.apps/docker-registry configured

Because we’re using a ClusterIP service, the IP address doesn’t change, which is convenient.

But, you’ll note, the previous ephemeral storage was discarded:

$ docker pull
Using default tag: latest
Error response from daemon: manifest for not found: manifest unknown: manifest unknown

Let’s see if what we’ve got now is better. We’ll test it by pushing a container…

$ docker pull hello-world
$ docker tag hello-world
$ docker push
$ docker pull

…deleting the pod…

$ kubectl --namespace docker-registry delete pod docker-registry-87bc44c4d-2w4zr
pod "docker-registry-87bc44c4d-2w4zr" deleted

…waiting for the replacement pod to start…

$ kubectl --namespace docker-registry get pods
NAME                              READY   STATUS    RESTARTS   AGE
docker-registry-87bc44c4d-stns9   1/1     Running   0          42s

…and then attempting to pull the image again:

$ docker pull
Using default tag: latest
Error response from daemon: manifest for not found: manifest unknown: manifest unknown

Oh, it … didn’t work.

What happened?

The problem is that the pod moved to a different node (comparing the before and after describe pod output…). So, while the pod has access to local storage on the node, it’s not the same storage, because it’s not the same node.

Where are my files?

If we used hostPath, these files have got to have gone somewhere, right? Our pod initially ran on rpi403, and our hostPath specified path: /tmp/docker-registry, so let’s go looking.

% ssh rpi403
$ ls -F /tmp
docker-registry/  ssh-flurble/  systemd-private-string-of-numbers/
$ cd /tmp/docker-registry
$ find . -type f

Yep. Those are our docker images.

If we look in the same place on the new node, we’ll find the /tmp/docker-registry directory, but it’s empty.

How to fix it?

We could fix this in a few different ways:

  • Use node affinity to always start the pod on the same node.
  • Use some kind of shared storage, such as NFS, so that all nodes have the same view of the volume.

Or we could use persistent volumes.