A few weeks ago, I decided to write a blog post about using mutual TLS to secure Erlang distribution (clustering), with auto-provisioning of certificates when running in Kubernetes. It took a little longer to write up than I expected, and turned into a series of blog posts.

For simplicity’s sake, I created a new application with rebar3 new app name=erlclu. I very soon regretted this decision, because I actually needed a release, so I ran rebar3 new release name=whoops and manually merged the relevant pieces together.

Because this is going to be a cluster of Erlang nodes, there’s (obviously) going to be more than one instance. It makes sense to add some kind of way to have some kind of “whoami” page, so that we can clearly see which node we’re talking to.

I noticed that whenever I made any change to the application, it caused the dockerpodman build to re-fetch and recompile all of the dependencies. On the tiny laptop I was using at the time, this was taking several extra minutes for every build.

I’ve covered this previously; see “Erlang/Elixir Cookies and Kubernetes”. Here’s the quick version.

Based on my investigation with libcluster in Elixir, I’ve decided to use DNS-based discovery for finding the other Erlang nodes in the cluster. To do this, we’ll need a headless service.

In the previous post, we got clustering working without TLS. Lifting from the investigation that I wrote up here, I’ll add TLS distribution to my Erlang cluster, but only with server certificates and with no verification (for now).

As explained here, I’m going to use an init container to issue the pod certificates.

Erlang TLS distribution is picky about how its certificates are created. OpenSSL is a pain to use. Here’s how to awkwardly wedge the two together. This took me several hours and a lot of my sanity. You’re welcome.

In the previous post we used OpenSSL to create a certificate signing request. In this post, we’ll submit it to cert-manager and get the certificate back.

In the previous two posts, we generated signing requests with OpenSSL and submitted them to cert-manager. In this post, we’ll actually use the generated certificates for mutual TLS.

As mentioned earlier, using TLS for Erlang distribution breaks erlclu remote_console (because it breaks erl_call). At the time, I worked around the problem by using nodetool. This post shows how to use Erlang’s SSH daemon instead.

In the previous post, I showed how to enable SSH access to the Erlang remote console on a pod. When we left it, it had no authentication. Let’s fix that.

In theory, we’ve got TLS working for our Erlang cluster, with mutual authentication. How do we prove that?

While scaling up/down the deployment for my Erlang cluster, I regularly refresh the web page that displays cluster members. Occasionally, I get a “502 Bad Gateway” error from Traefik. What’s with that?

The Erlang daemon that I’ve been using to demonstrate Erlang distribution on K8s is running as the root user. I’m going to fix that.

Erlang doesn’t like running as pid 0 (usually the init process). I used tini to fix it.

In an earlier post, I used a ClusterIssuer that I originally created when first setting up cert-manager. That needs fixing.

In a previous post, I used sleep 5s to wait for cert-manager to complete the CertificateRequest. Instead, we should poll the status field.

In this post, I showed how to use an init container to create CertificateRequest objects, which cert-manager signs, returning the certificates. A new request is created every time a pod starts. This eventually leaves a lot of stale CertificateRequest objects. We should clean those up.

This started as a quick experiment to spin up an Erlang cluster on Kubernetes, using TLS distribution, and to validate using cert-manager to issue the pod certificates.