Jetstack Cert-Manager

One of my least favorite parts of computers is dealing with certificate creation. In fact, ya know those tweets about what you’d tweet if you were kidnapped and didn’t want to tip off the kidnapers?

Yeah, I’d tweet about how I love working with certificates. They are just not a fun thing for me. So when I found a new project where I needed certificates created, I was not really excited.

What I found though, was a neat little project called jetstack cert-manager, that will automatically create and apply certificates to Kubernetes resources and it was really simple to setup and use.

Overview

Cert-manager is a Kubernetes add on that works with a variety of issuers including Hashicorp Vault and LetsEncrypt Certificate Authorities. You can use this to create your own certificates by issuing kubectl commands, but an even more powerful way to use it is to have it automatically create certificates when an ingress resource is deployed.

When a new ingress resources is created to pass traffic through a reverse proxy, cert-manager responds by requesting a new certificate and validates it with the CA, then applies the certificate to secure your traffic. So in a sense, you can set this up one time, and then have all your web servers protected with an SSL cert just by adding a couple of lines to your ingress resources.

Let’s see it in action in my AWS cluster using the LetsEncrypt. We’ll deploy cert-manager and issue a staging cert which is a temporary cert to make sure things are working properly, and then later a production certificate which is fully trusted by most browsers.

Prerequisites

Before we deploy cert-manager, let’s get some basics setup and configured. We’ll need an ingress resource that is available on the Internet so that LetsEncrypt can issue the http-01 or dns-01 challenges to validate the domain names. We’ll also need an ingress controller and obviously a Kubernetes cluster with some capacity in it for our resources.

Here is a snapshot of the lab I’ll be working with:

Install Cert-Manager

The installation of cert-manager is pretty simple. We need to apply a couple of manifests.

First we’ll create a new namespace where the cert-manager resources will live.

kubectl create namespace cert-managerCode language: PHP (php)

Next up, its time to deploy cert-manager.

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.11.0/cert-manager.yamlCode language: JavaScript (javascript)

I don’t want to minimize the work we’ve done here, but the deployment is pretty much done. You can see if everything looks ok by running:

kubectl get all -namespace cert-managerCode language: JavaScript (javascript)

There should be a list of pods and services similar the the example below.

Configure Cert-Manager

Now we need to configure cert-manager to issue new certificates. We do this by deploying issuers. There are two type of issuers: Issuers and ClusterIssuers. Just like with Roles and ClusterRoles, the difference is the scope in which they work. Since I want to be able to share this service across namespaces, I’ll be deploying cluster issuers. The ClusterIssuer used in my cluster is shown below. You may notice that there are two different issuers being deployed. We’ll use staging first which has a more lenient rate limit, and production when we’ve got it working.

apiVersion: cert-manager.io/v1alpha2 #v11
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    email: user@domain.com #use your own email address - change as needed
    solvers:
      - selector: {}
        http01:
          ingress:
            class: contour #ingress controller in use - change as needed
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory #letsencrypt url
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
  namespace: cert-manager
spec:
  acme:
    email: user@domain.com #use your own email address - change as needed
    solvers:
      - selector: {}
        http01:
          ingress:
            class: contour #ingress controller in use - change as needed
    privateKeySecretRef:
      name: letsencrypt-production 
    server: https://acme-v02.api.letsencrypt.org/directory #letsencrypt url

Deploy the manifest with the apply command:

kubectl apply -f [manifest file].yaml

Once deployed, you should be able to query them through kubectl.

If you want to go one level deeper, run a kubectl describe on the ClusterIssuers. You should see that the account was registered correctly.

Deploy Staging App

In this part of the post, I’m deploying a custom app of mine that has a backend database and things that aren’t shown here. However, the app pieces are shown below and your own app should work with the cert-manager additions.

In this phase we’ll first deploy our web app and make sure we can get a certificate installed, even if it isn’t a trusted cert.

Here is the app manifest I’m deploying, complete with the ingress rules needed for cert-manager.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hollowapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hollowapp
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: hollowapp
    spec:
      containers:
        - name: hollowapp
          image: eshanks16/hollowapp:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 5000
          env:
            - name: SECRET_KEY
              value: "my-secret-key"
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: hollow-secret
                  key: db.string
---
apiVersion: v1
kind: Service
metadata:
  name: hollowapp
  labels:
    app: hollowapp
spec:
  type: ClusterIP
  ports:
    - port: 5000
      protocol: TCP
      targetPort: 5000
  selector:
    app: hollowapp
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hollowapp
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-staging" #Name of the cluster issuer goes here!!!!!
    kubernetes.io/ingress.class: contour #ingress controller in use
    kubernetes.io/tls-acme: "true" #acme cert being requested
  labels:
    app: hollowapp
spec:
  tls: #TLS Spec listed here. 
  - hosts: 
    - hollowapp.theithollowlab.com #External DNS Name 
    secretName: hollowapp-tls-stage 
  rules:
    - host: hollowapp.theithollowlab.com
      http:
        paths:
        - path: /
          backend:
            serviceName: hollowapp
            servicePort: 5000

The main pieces of the above manifest are:

cert-manager.io/cluster-issuer: “letsencrypt-staging” – specify the ClusterIssuer

kubernetes.io/ingress.class: contour – change based on your ingress controller in use.

host: hollowapp.theithollowlab.com – change to your own DNS name.

secretName: hollowapp-tls-stage – a secret name meaningful to you. This is where your app will store the certificate returned by cert-manager.

Apply your app manifest and wait a moment or two before checking the https://URL. It does take a little time for LetsEncrypt to validate the VM and provide the appropriate certificate information. You can see that my https site came up, although the certificate is not valid, as seen by the “Not Secure” moniker.

Now that our test certificate worked, let’s change the issuer requested in our ingress manifest to use the production ClusterIssuer instead of staging. Re-apply that manifest and again wait a moment before the certificates are applied. Here we can see that a fully trusted certificate was automatically applied to my app.

Summary

Jetstack cert-manager is a pretty simple solution to install in your Kubernetes cluster that will automatically add new certificates for your ingress resources. The ingress resource gets applied with the proper specifications and cert-manager does the rest, to give you a fully trusted certificate. What exactly will you do with all the time you got back from not having to do certificate requests over and over again?