If you want to replace an Erlang module on a remote node, but you’re unable to copy files to it, and you’re unable to make RPC calls to it, but you can get a remote console, here’s what you do:

Erlang’s gen_server (and gen_statem, etc.) allow for you to pass {debug, DebugOptions} in gen_server:start_link/4 (et al.). The documentation for it is spread over several pages, making it hard to understand at first. How should we use it?

In a previous post, I showed how to store sys.config and vm.args in a ConfigMap. There are some problems with that approach, so I’m going to use Kustomize to fix that.

Here’s how you can use via in a name specification when starting a gen_server (or gen_statem, etc.) to register your process with a process registry.

You can gen_server:call in a bunch of different ways.

Erlang configuration is typically stored in the sys.config file. What options do we have if we want to have different settings in this file when deploying using Kubernetes?

About 3 months ago, the Erlang nodes in my cluster stopped talking to each other. This was caused by the CA certificate expiring. In that post, I asked “How do we roll the CA certificate without downtime?”. This post explores some options.

In several of the previous posts, for example Using ‘setup’, I’ve started a server in suite_setup/0 and needed to kill it in suite_cleanup/1. I showed a simple way to do that, but it’s not the best. Here’s a better way to do it.

Similar to passing the result from {setup, Setup, ... to each test, you can also pass the result from {foreach, Setup, ... to each test, but there are some differences.

The result from setup is passed to cleanup already. What if we want to pass it to each of our tests? It’s a bit awkward, but it looks like the following:

If you want to run some setup before (and cleanup after) a list of tests, and also some setup and cleanup for each test, you can nest setup and foreach:

By default, Erlang distribution listens on all available network interfaces. Sometimes you don’t want this. How can we fix it?

We saw that setup runs setup/cleanup before all tests in a list. If you want to run some setup/cleanup before each test in a list, you can use foreach:

What if we want to run some setup before a set of tests (and cleanup afterwards)?

While you could specify each test with a single function, you can also use a test “instantiator” (or generator) and have it return a list.

A simple EUnit test might look like this:

Some examples of how to use Erlang’s unit-testing framework, EUnit. Starting with simple examples and getting more advanced.

Sometimes, you’re connected to another node in an Erlang cluster, and you need to inject a new or replacement module. Here’s how.

Sometimes, when you run rebar3 shell and you press Ctrl+C twice to quit, your terminal is messed up. This is how I fixed it in zsh.

In the previous post, I wrote about customizing the Erlang shell prompt by using the .erlang file. Erlang looks for this file in $HOME (I’m simplifying here); it doesn’t look in the current working directory. How can we fix that?

Did you know that you can change the Erlang shell prompt?

How do I change the title of my terminal window from Erlang?

How do I encrypt (and decrypt) things in Erlang, using an RSA key?

In an earlier post, I showed how to encrypt a JSON Web Token in Erlang. I used plain-ol’ Erlang. Here’s how to use the jose package to do the same thing.

Following on from signing a web token in Erlang, here’s how to encrypt one.

In an earlier post, I showed how to sign a JSON Web Token in Erlang. I used plain-ol’ Erlang. Here’s how to use the jose package to do the same thing.

About 8 years ago, I wrote a post about verifying JSON Web Tokens (JWT) in Erlang. 3 years later, I wrote another post about signing a JWT in bash. This is a post combining the two: here’s how you sign a JWT using Erlang.

How do I sign (and verify) things in Erlang, using an RSA key?

The previous post finished up by writing an RSA public key in “wrapped” (PKCS#8) PEM-format. It occurred to me that I should probably show how to do the same with the private key as well.

I’m currently doing something that needs RSA keys, so here are some Erlang snippets showing how to create one.

A question on Mastodon asks “What are people using [for cluster management] in 2023?”. I thought I’d address a couple of hidden assumptions in the question and do a quick survey of what’s available.

By default, the improved logger originally introduced in Erlang/OTP-21 doesn’t support per-level colours. This is something that I miss from lager and from Elixir’s Logger. Here’s a simple way to implement something like lager.

In the previous posts, we gathered metrics from Cowboy and Hackney. I’d like to publish the metrics to Prometheus.

In the previous post of the series, I added basic metrics reporting to Cowboy, and simply wrote them to the logger. In this post, I’m going to do the same for the HTTP client, Hackney,.

Cowboy is probably the most popular HTTP server for the Erlang and Elixir ecosystem. Here’s how to get metrics from it.

Building and installing Erlang is almost compatible with my use erlang rule for direnv. Here’s how to bodge it.

There’s more comprehensive information in various files in the HOWTO directory, but ain’t nobody got time for that. This works for me.

Some notes about hacking on and contributing to Erlang, because I don’t do it frequently enough to have all of this in muscle memory.

You’re experimenting with Erlang’s built-in SSH daemon, and it fails with “No host key available”. What’s up with that?

The nodes in the cluster stopped talking to each other at some point. I only noticed this afternoon after investigating some other problem.

For when you want to run both per-suite and per-test setup and cleanup in eunit tests.

This started as a quick experiment to spin up an Erlang cluster on Kubernetes, using TLS distribution, and to validate using cert-manager to issue the pod certificates.

While scaling up/down the deployment for my Erlang cluster, I regularly refresh the web page that displays cluster members. Occasionally, I get a “502 Bad Gateway” error from Traefik. What’s with that?

In theory, we’ve got TLS working for our Erlang cluster, with mutual authentication. How do we prove that?

In the previous post, I showed how to enable SSH access to the Erlang remote console on a pod. When we left it, it had no authentication. Let’s fix that.

As mentioned earlier, using TLS for Erlang distribution breaks erlclu remote_console (because it breaks erl_call). At the time, I worked around the problem by using nodetool. This post shows how to use Erlang’s SSH daemon instead.

When you’re investigating a problem with a deployed application, it’s useful to know precisely which version you’re looking at. Here’s how to automatically set the version number in an Erlang project.

In the previous two posts, we generated signing requests with OpenSSL and submitted them to cert-manager. In this post, we’ll actually use the generated certificates for mutual TLS.

As explained here, I’m going to use an init container to issue the pod certificates.

In the previous post, we got clustering working without TLS. Lifting from the investigation that I wrote up here, I’ll add TLS distribution to my Erlang cluster, but only with server certificates and with no verification (for now).

Based on my investigation with libcluster in Elixir, I’ve decided to use DNS-based discovery for finding the other Erlang nodes in the cluster. To do this, we’ll need a headless service.

I’ve covered this previously; see “Erlang/Elixir Cookies and Kubernetes”. Here’s the quick version.

I noticed that whenever I made any change to the application, it caused the dockerpodman build to re-fetch and recompile all of the dependencies. On the tiny laptop I was using at the time, this was taking several extra minutes for every build.

Because this is going to be a cluster of Erlang nodes, there’s (obviously) going to be more than one instance. It makes sense to add some kind of way to have some kind of “whoami” page, so that we can clearly see which node we’re talking to.

For simplicity’s sake, I created a new application with rebar3 new app name=erlclu. I very soon regretted this decision, because I actually needed a release, so I ran rebar3 new release name=whoops and manually merged the relevant pieces together.

A few weeks ago, I decided to write a blog post about using mutual TLS to secure Erlang distribution (clustering), with auto-provisioning of certificates when running in Kubernetes. It took a little longer to write up than I expected, and turned into a series of blog posts.

Some notes about using rebar3 with an umbrella project.

In the previous post, I recapped Erlang distribution (clustering). In this post, we’ll secure it by using TLS.

I want to write a post about using mutual TLS to secure Erlang distribution (clustering), with auto-provisioning of certificates when running in Kubernetes. This is not that post. This is a recap of basic Erlang clustering, to refresh my memory and lay some groundwork.

Erlang/OTP provides a built-in SSH client and daemon. You can use this to expose the console directly over SSH.

You’ve got an Erlang module with a private (not-exported) function, and you want to add some unit tests for that function? How should you do that?

The ErlangLS extension for VS Code includes formatting, using rebar3_format. I’d prefer to use erlfmt, so here’s how I set it up.

Distributed Erlang and Elixir applications use a shared secret called a “cookie”. It’s just a string of alphanumeric characters. All nodes in the cluster must use the same cookie. Let’s take a look at what that means in a Kubernetes context.

To get the version of the compiler used to compile a particular .beam file:

Erlang/OTP 24.0 added support for ed25519 curves. Here’s some example snippets:

I recently had to report a bug against ErlangLS (the Erlang Language Server). Here’s how I discovered the version number:

Per Wikipedia:

Ordinarily when writing an SSL/TLS server or client using Erlang/OTP, you’ll use the certfile and keyfile options, as follows:

I’ve just spent about a day poking around in the guts of Erlang/OTP and Ranch, and I thought I’d write some of it down.

At Electric Imp (now part of Twilio), my team uses Erlang’s Common Test for driving our system tests. These are (almost-)end-to-end tests that exercise (almost) the whole platform.

I’ve got a bunch of Erlang nodes running in Docker containers, and I’d like to connect a remote shell, running on the host, to one of them.

Quick reference for installing Erlang and Elixir on a Raspberry Pi, using the Erlang Solutions packages.

When an error occurs in the Electric Imp backend, it’s logged and collated, and we regularly review error reports to see if there’s anything that needs looking at.

Use this helper function:

At Electric Imp, on developer PCs, we manage our Erlang versions with kerl.

To integrate direnv with kerl, add the following to ~/.direnvrc:

Ubuntu

If you want to use kerl to build your Erlang installation, you’re going to need some packages installed first.

If kerl list installations is displaying Erlang installations that you deleted ages ago, and you’ve got all of your installations in ~/.kerl/erlangs/, you can rebuild the list by running the following command:

When you use gen_server:start/3,4 or gen_server:start_link/3,4, the call blocks until the other process has finished running init/1.

Hello World

I wanted to get Erlang running on my Synology NAS, for various reasons, and was struggling with the cross compiler.

I’m currently playing with OpenID Connect (OAuth 2.0 for Login), to allow people to log into a web site using their Google account. The web site is built using Erlang.

We use lager for our logging at Electric Imp. This morning I had cause to tweak the configuration at runtime on one of our staging boxes, but first I needed to figure out which handlers were already installed.

In Erlang, in a gen_server, when does terminate get called? Also, some messing around with dbg for tracing.

I find Erlang’s gen_event behaviour to be fairly tricky to understand, despite the copious documentation on the subject:

It turns out that you can’t load Erlang NIF libraries from the shell.

kerl allows you to easily build and install multiple Erlang/OTP releases. It’s kinda like nvm or rvm, but for Erlang. It doesn’t do everything, and that’s what direnv is for. direnv allows you to run commands upon entering or leaving a particular directory.

While using erlang.mk and relx to build a newly-created Erlang application, I got the following error:

Originally posted to the Electric Imp blog. Preserved here.

Overview

You’re using lager for logging in your Erlang program, and you discover that your configuration isn’t logging anything more verbose than info-level messages. How do you bump that up to debug?

If you’re connected to an Erlang node via a remote shell, and you don’t have access to stdout on the original node, you’ll need to redirect the trace output to your current shell.

It’s pretty simple to create a ZIP file on disk.

Some odds and ends.

handle_event and handle_sync_event

Background

Just some random thing I learnt today while playing with Erlang: