Cert-Manager Audit

In Expired Certificates: Incident Review, I listed a future action: “Audit the cluster to see if there are any other TLS secrets that aren’t using cert-manager.” Here’s how I did it using Elixir Livebook.

Cert-manager metadata

A secret that’s been issued by cert-manager has various annotations. Here’s my gitea-tls secret (ellided, obviously):

% kubectl --namespace gitea get secret gitea-tls -o yaml

apiVersion: v1
data:
  ca.crt: ...
  tls.crt: ...
  tls.key: ...
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: git.k3s.differentpla.net
    cert-manager.io/certificate-name: git-k3s-differentpla-net
    cert-manager.io/common-name: ""
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: k3s-ca-cluster-issuer
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2023-03-18T20:06:04Z"
  name: gitea-tls
  namespace: gitea
  resourceVersion: "25878364"
  uid: f51be88d-503e-46ff-8133-48be6ba09f6b
type: kubernetes.io/tls

So that’s how I’ll do it:

1. Get all the secrets with type: kubernetes.io/tls.
2. Ignore the ones with cert-manager annotations.

If all of the TLS secrets are managed by cert-manager, the resulting list should be empty, right?

Well, no, there are other system-issued TLS secrets, but we can also ignore those.

Install dependencies

Mix.install([{:k8s, "~> 1.2"}])

Connect

{:ok, conn} = K8s.Conn.from_service_account()
:ok

Note that my Livebook installation has cluster admin access.

List TLS secrets

op = K8s.Client.list("v1", "Secret")
{:ok, secrets} = K8s.Client.run(conn, op)

tls_secrets =
  secrets["items"]
  |> Enum.filter(fn
    %{"type" => "kubernetes.io/tls"} -> true
    _ -> false
  end)

Filter functions

issued_by_cert_manager? = fn
  %{"metadata" => %{"annotations" => %{"cert-manager.io/issuer-name" => _}}} -> true
  _ -> false
end

is_in_namespace? = fn %{"metadata" => %{"namespace" => namespace}}, n -> namespace == n end
is_named? = fn %{"metadata" => %{"name" => name}}, n -> name == n end

Run the query

tls_secrets
|> Enum.reject(fn secret ->
  is_in_namespace?.(secret, "kube-system") or is_in_namespace?.(secret, "cert-manager") or
    is_in_namespace?.(secret, "longhorn-system")
end)
|> Enum.reject(&is_named?.(&1, "erlclu-ca-key-pair"))
|> Enum.reject(issued_by_cert_manager?)

We ignore system-issued TLS secrets, those in the cert-manager namespace itself, and also those managed by Longhorn. The CA keypair for my Erlang cluster is also manually-managed.

That first Enum.reject could probably be shortened by using Enum.any?/2, but I’m suffering from a cold at the moment, so my brain’s not really working properly.

The query should return an empty list.