k3s on Raspberry Pi: Using an Insecure Docker Registry
Let’s see if we can push an image to our new Docker Registry.
Installing Docker CLI
First we need to install the Docker CLI.
Because this is Raspbian, we need to use the “convenience” script that Docker provides. See Install using the convenience script.
Like it says:
Tip: preview script steps before running
curl -fsSL https://get.docker.com -o get-docker.sh
DRY_RUN=1 sh ./get-docker.sh
Happy? Then we can run it:
sudo sh get-docker.sh
Pull “hello-world”
We’re going to need something to push, so we’ll just use the “hello-world” image:
$ sudo docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
9b157615502d: Pull complete
Digest: sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest
docker permissions
That sudo docker stuff is going to get old quickly. Let’s sort out permissions:
$ sudo usermod -aG docker pi
$ exit # and log back in
Note that if you’re using Ubuntu, systemd will re-use your session when you log back in. You basically need to reboot to sort it out.
Fortunately, we’re not on Ubuntu; we’re accessing the node over SSH, so
a simple Ctrl+D, Up, Enter gets us logged back in.
Pushing an image
$ docker tag hello-world 10.43.236.176:5000/hello-world
$ docker push 10.43.236.176:5000/hello-world
Using default tag: latest
The push refers to repository [10.43.236.176:5000/hello-world]
Get "https://10.43.236.176:5000/v2/": http: server gave HTTP response to HTTPS client
Ah. That’s a problem. Can we tell docker to not use HTTPS for now?
Configuring docker to use an insecure registry
Yes, you can. See this Stack Overflow question.
Create (or edit) /etc/docker/daemon.json, as follows:
{"insecure-registries": ["10.43.236.176:5000"]}
Then restart the docker daemon:
sudo systemctl restart docker
$ sudo docker push 10.43.236.176:5000/hello-world
Using default tag: latest
The push refers to repository [10.43.236.176:5000/hello-world]
a380e59f1cab: Pushed
latest: digest: sha256:f130bd2d67e6e9280ac6d0a6c83857bfaf70234e8ef4236876eccfbd30973b1c size: 525
$ sudo docker run 10.43.236.176:5000/hello-world
Hello from Docker!
...
Using an image
At this point, we should be able to run a container with that image, so let’s create hello-world.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
namespace: docker-registry
labels:
app: hello-world
spec:
replicas: 1
selector:
matchLabels:
app: hello-world
template:
metadata:
labels:
app: hello-world
name: hello-world
spec:
containers:
- name: hello-world
image: 10.43.236.176:5000/hello-world
$ kubectl apply -f hello-world.yml
deployment.apps/hello-world created
$ kubectl --namespace docker-registry get all
NAME READY STATUS RESTARTS AGE
pod/docker-registry-684dc65c99-pckxl 1/1 Running 0 137m
pod/hello-world-5449677b5b-96crm 0/1 ErrImagePull 0 6s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/docker-registry ClusterIP 10.43.236.176 <none> 5000/TCP 106m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/docker-registry 1/1 1 1 137m
deployment.apps/hello-world 0/1 1 0 7s
NAME DESIRED CURRENT READY AGE
replicaset.apps/docker-registry-684dc65c99 1 1 1 137m
replicaset.apps/hello-world-5449677b5b 1 1 0 6s
Hmmm. ErrImagePull doesn’t look good. Let’s investigate.
$ kubectl --namespace docker-registry describe pod hello-world-5449677b5b-96crm
Name: hello-world-5449677b5b-96crm
Namespace: docker-registry
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 94s default-scheduler Successfully assigned docker-registry/hello-world-5449677b5b-96crm to rpi403
Normal BackOff 26s (x4 over 93s) kubelet Back-off pulling image "10.43.236.176:5000/hello-world"
Warning Failed 26s (x4 over 93s) kubelet Error: ImagePullBackOff
Normal Pulling 12s (x4 over 93s) kubelet Pulling image "10.43.236.176:5000/hello-world"
Warning Failed 12s (x4 over 93s) kubelet Failed to pull image "10.43.236.176:5000/hello-world": rpc error: code = Unknown desc = failed to pull and unpack image "10.43.236.176:5000/hello-world:latest": failed to resolve reference "10.43.236.176:5000/hello-world:latest": failed to do request: Head "https://10.43.236.176:5000/v2/hello-world/manifests/latest": http: server gave HTTP response to HTTPS client
Warning Failed 12s (x4 over 93s) kubelet Error: ErrImagePull
Yeah, we get the same server gave HTTP response to HTTPS client error
as before.
Let’s clean up:
kubectl --namespace docker-registry delete deployment hello-world
Configuring k3s containerd to use an insecure registry
To fix that, we need to configure containerd. For k3s, that’s documented here.
We need to create a file /etc/rancher/k3s/registries.yaml on each node,
as below. Note that the directory might not exist on the worker nodes:
sudo mkdir -p /etc/rancher/k3s/
sudo chmod 755 /etc/rancher/k3s/
sudo vi /etc/rancher/k3s/registries.yaml # as follows
mirrors:
"10.43.236.176:5000":
endpoint:
- "http://10.43.236.176:5000"
The mirror name is the name of our registry. For now, I’m going to use the dotted-IP, because I don’t want to get into naming things until later.
I’m probably going to run into problems because this is a ClusterIP, but this is all about the learning.
The endpoint uses http://, which disables the use of TLS.
According to the documentation, we can add multiple endpoints, which seems like a reasonable way to avoid needing a load balancer.
This file needs to be deployed on all nodes where it will be used, and the k3s service needs to be restarted:
sudo systemctl restart k3s # on the master
sudo systemctl restart k3s-agent # on the workers
Obviously, if you were using this in production, you’d migrate the load off the workers, and you’d be using HA master-only nodes, right?
Equally obviously, if you were using this in production, you wouldn’t be avoiding TLS, and you’d have working DNS…
Try it again
$ kubectl apply -f hello-world.yml
deployment.apps/hello-world created
$ kubectl --namespace docker-registry get all
NAME READY STATUS RESTARTS AGE
pod/docker-registry-684dc65c99-pckxl 1/1 Running 0 6d5h
pod/hello-world-5449677b5b-7wwtm 0/1 CrashLoopBackOff 1 20s
Ooh. CrashLoopBackOff. That’s different. Is that good?
$ kubectl --namespace docker-registry describe pod/hello-world-5449677b5b-7wwtm
Name: hello-world-5449677b5b-7wwtm
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 61s default-scheduler Successfully assigned docker-registry/hello-world-5449677b5b-7wwtm to rpi404
Normal Pulled 60s kubelet Successfully pulled image "10.43.236.176:5000/hello-world" in 543.718498ms
Normal Pulled 59s kubelet Successfully pulled image "10.43.236.176:5000/hello-world" in 250.715721ms
Normal Pulled 42s kubelet Successfully pulled image "10.43.236.176:5000/hello-world" in 190.243966ms
Normal Pulling 18s (x4 over 61s) kubelet Pulling image "10.43.236.176:5000/hello-world"
Normal Pulled 18s kubelet Successfully pulled image "10.43.236.176:5000/hello-world" in 203.508818ms
Normal Created 18s (x4 over 60s) kubelet Created container hello-world
Normal Started 17s (x4 over 59s) kubelet Started container hello-world
Warning BackOff 3s (x6 over 58s) kubelet Back-off restarting failed container
This bit right here: Successfully pulled image "10.43.236.176:5000/hello-world" in 203.508818ms. Yes, it’s good.
We still need to clean up, though:
$ kubectl --namespace docker-registry delete deployment hello-world
Recap
So far, we’ve got a private docker registry running in a pod, and we can use it to provide images to our nodes.
But there are problems, and they all come down to the same two things: DNS and load-balancing.
- We’re using ClusterIP, so we can’t push images from outside the cluster.
- If we add a NodePort service:
- The name changes if the pod moves to a different node.
- We can deal with that for external pushes if we’re prepared to look up the IP address each time.
- We really ought to be using TLS.
- But we can’t do that if the name’s going to keep changing.
- …or if it’s got two different names. I can’t be bothering with SNI.
- Docker image tags have the registry name in them.
- Currently this is the dotted IP address.
- We can use a fixed mirror name in
registries.yamlto deal with that. - But that won’t work outside the cluster.
- If the registry name changes, does that break the image tags?
We can work around these problems for a while, but at some point we’ll need to bite the bullet. That moment is getting closer, but we’ve got a few other things to poke at first.