Erlang cluster on Kubernetes: Is it mutual?

22 Dec 2022 16:45 erlang

In theory, we’ve got TLS working for our Erlang cluster, with mutual authentication. How do we prove that?

Here’s one way. It’s a complete hack. It relies on internal details of the Erlang runtime. Don’t try this at home.

Listing connections

If you run inet:i(). in the Erlang console, you’ll see a table of all of the active TCP sockets in the process. It looks like this:

(erlclu@> inet:i().
Port Module   Recv  Sent    Owner     Local Address    Foreign Address  State        Type
24   inet_tcp 0     0       <0.489.0> *:46025          *:*              ACCEPTING    STREAM
40   inet_tcp 6     21      <0.487.0> localhost:40110  localhost:epmd   CONNECTED(O) STREAM
64   inet_tcp 0     0       <0.570.0> *:22             *:*              ACCEPTING    STREAM
72   inet_tcp 0     0       <0.573.0> *:http-alt       *:*              ACCEPTING    STREAM
80   inet_tcp 0     0       <0.586.0> *:9153           *:*              ACCEPTING    STREAM
216  inet_tcp 4591  4914    <0.663.0> CONNECTED(O) STREAM
232  inet_tcp 4589  4915    <0.675.0> CONNECTED(O) STREAM
296  inet_tcp 4609  4358    <0.693.0> CONNECTED(O) STREAM
320  inet_tcp 3354  3793    <0.701.0> CONNECTED(O) STREAM
368  inet_tcp 3247  1524923 <0.710.0> CONNECTED(O) STREAM
648  inet_tcp 19472 73069   <0.740.0> localhost:22     localhost:36766  CONNECTED(O) STREAM

There are a number of different connections in there. We can identify them as follows:

  • The ones on port 22 are the SSH daemon and the current remote console connection.
  • The one listening on http-alt is the cowboy endpoint.
  • The one connected to epmd is, well, connected to epmd.
  • The one listening on 9153 is the metrics endpoint (the subject of a future blog post).
  • This means that the one listening on port 46025 is probably the TLS distribution endpoint.
  • The others have port 46025 in the local address (TLS server) or in the remote address (TLS client).

We can confirm the distribution port by asking epmd:

(erlclu@> erl_epmd:names().

Yep. Port 46025 it is.

So all we need to do is – somehow – convert the owner into something that we can pass to ssl:connection_information/1, and we’ll be able to ask questions about that connection, including whether it’s using mutual authentication.

The hack

Here’s how to turn that owner pid into an #sslsocket{} record, suitable for passing to ssl:connection_information.


% You've got a TLS socket in inet:i(), and you want to use
% ssl:connection_information with it.
sslsocket_from_pid(Pid) ->
    {connection, State} = sys:get_state(Pid),
    StaticEnv = element(2, State),
    Port = element(11, StaticEnv),
    ProtocolSpecific = element(9, State),
    Pid2 = maps:get(sender, ProtocolSpecific),

    Transport = gen_tcp,
    ConnectionCb = tls_connection,
    {sslsocket, {Transport, Port, ConnectionCb, undefined}, [Pid, Pid2]}.
It’s tightly-coupled to the implementation details. I’ve only tested it on Erlang/OTP-25.1.2. Basically: don’t do this.

Client authentication

We assume that server verification works. We’re interested in whether we’re using client certificates for authentication.

We need to look at the connections which have us as the server. Specifically, these two:

296  inet_tcp 4609  4358    <0.693.0> CONNECTED(O) STREAM
320  inet_tcp 3354  3793    <0.701.0> CONNECTED(O) STREAM

By converting one of those into an sslsocket, we can get the peer cert:

(erlclu@> S = erlclu_inspect:sslsocket_from_pid(pid(0,701,0)).
(erlclu@> {ok, Cert} = ssl:peercert(S).
(erlclu@> public_key:pkix_decode_cert(Cert, otp).

It has a certificate. That means it’s using mutual TLS. Job done.