Kubernetes Validating Admission Controllers

Hey! Who deployed this container in our shared Kubernetes cluster without putting resource limits on it? Why don’t we have any labels on these containers so we can report for charge back purposes? Who allowed this image to be used in our production cluster?

If any of the questions above sound familiar, its probably time to learn about Validating Admission Controllers.

Validating Admission Controllers – The Theory

Admission Controllers are used as a roadblocks before objects are deployed to a Kubernetes cluster. The examples from the section above are common rules that companies might want to enforce before objects get pushed into a production Kubernetes cluster. These admission controllers can be from custom code that you’ve written yourself, or a third party admission controller. A common open-source project that manages admission control rules is Open Policy Agent (OPA).

A generalized request flow for new objects starts with Authenticating with the API, then being authorized, and then optionally hitting an admission controller, before finally being committed to the etcd store.

The great part about an admission controller is that you’ve got an opportunity to put our own custom logic in as a gate for API calls. This can be done in two forms.

ValidatingAdmissionController – This type of admission controller returns a binary result. Essentially, this means either yes the object is permitted or no the object is not permittetd.

MutatingAdmissionController – This type of admission controller has the option of modifying the API call and replacing it with a different version.

As an example, our example requirement that a label needs to be added to all pods, could be handled by either of these methods. A validating admission controller would allow or deny a pod from being deployed without the specified tag. Meanwhile, in a mutating admission controller, we could add a tag if one was missing, and then approve it. This post focuses on building a validating admission controller.

Once the API request hits our admission controller webhook, it will make a REST call to the admission controller. Next, the admission controller will apply some logic on the Request, and then make a response call back to the API server with the results.

The api request to the admission controller should look similar to the request below. This was lifted directly from the v1.18 Kubernetes documentation site.

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "request": {
    # Random uid uniquely identifying this admission call
    "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",

    # Fully-qualified group/version/kind of the incoming object
    "kind": {"group":"autoscaling","version":"v1","kind":"Scale"},
    # Fully-qualified group/version/kind of the resource being modified
    "resource": {"group":"apps","version":"v1","resource":"deployments"},
    # subresource, if the request is to a subresource
    "subResource": "scale",

    # Fully-qualified group/version/kind of the incoming object in the original request to the API server.
    # This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the
    # original request to the API server was converted to a version the webhook registered for.
    "requestKind": {"group":"autoscaling","version":"v1","kind":"Scale"},
    # Fully-qualified group/version/kind of the resource being modified in the original request to the API server.
    # This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the
    # original request to the API server was converted to a version the webhook registered for.
    "requestResource": {"group":"apps","version":"v1","resource":"deployments"},
    # subresource, if the request is to a subresource
    # This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the
    # original request to the API server was converted to a version the webhook registered for.
    "requestSubResource": "scale",

    # Name of the resource being modified
    "name": "my-deployment",
    # Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object)
    "namespace": "my-namespace",

    # operation can be CREATE, UPDATE, DELETE, or CONNECT
    "operation": "UPDATE",

    "userInfo": {
      # Username of the authenticated user making the request to the API server
      "username": "admin",
      # UID of the authenticated user making the request to the API server
      "uid": "014fbff9a07c",
      # Group memberships of the authenticated user making the request to the API server
      "groups": ["system:authenticated","my-admin-group"],
      # Arbitrary extra info associated with the user making the request to the API server.
      # This is populated by the API server authentication layer and should be included
      # if any SubjectAccessReview checks are performed by the webhook.
      "extra": {
        "some-key":["some-value1", "some-value2"]
      }
    },

    # object is the new object being admitted.
    # It is null for DELETE operations.
    "object": {"apiVersion":"autoscaling/v1","kind":"Scale",...},
    # oldObject is the existing object.
    # It is null for CREATE and CONNECT operations.
    "oldObject": {"apiVersion":"autoscaling/v1","kind":"Scale",...},
    # options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions.
    # It is null for CONNECT operations.
    "options": {"apiVersion":"meta.k8s.io/v1","kind":"UpdateOptions",...},

    # dryRun indicates the API request is running in dry run mode and will not be persisted.
    # Webhooks with side effects should avoid actuating those side effects when dryRun is true.
    # See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details.
    "dryRun": false
  }
}Code language: PHP (php)

The response REST call back to the API server should look similar to the response below. The uid must match the uid from the request in v1 of the admission.k8s.io api. The allowed field is true for permitting the request to go through and false if it should be denied.

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<value from request.uid>",
    "allowed": true/false
  }
}Code language: JSON / JSON with Comments (json)

Validating Admission Controllers – In Action

Now, let’s build our own custom validating admission controller in python using a Flask API. The goal of this controller is to ensure that all deployments and pods have a “Billing” label added. If you’d like to get a headstart with the code presented in this post, you may want to pull down the github repo: https://github.com/theITHollow/warden

We’ll assume that we’re doing some chargeback/showback to our customers and we need this label on everything or we can’t identify what customer it belongs to, and we can’t bill them.

Create the Admission Controller API

The first step we’ll go through is to build our flask API and put in our custom logic. If you look in the flask API example below, I’m handling an incoming request to the /validate URI and checking the metadata of the object for a “billing” label. I’m also capturing the uid of the request so I can pass that back in the response. Also, be sure to provide a message so that the person requesting the object gets feedback about why the operation was not permitted. Then depending on the label being found, a response is created with an allowed value of True or False.

This is a very simple example, but once you’ve set this up, use your own custom logic.

from flask import Flask, request, jsonify

warden = Flask(__name__)
 
#POST route for Admission Controller  
@warden.route('/validate', methods=['POST'])

#Admission Control Logic
def deployment_webhook():
    request_info = request.get_json()
    uid = request_info["request"].get("uid")
    try:
        if request_info["request"]["object"]["metadata"]["labels"].get("billing"):
            #Send response back to controller if validations succeeds
            return k8s_response(True, uid, "Billing label exists")
    except:
        return k8s_response(False, uid, "No labels exist. A Billing label is required")
    
    #Send response back to controller if failed
    return k8s_response(False, uid, "Not allowed without a billing label")

#Function to respond back to the Admission Controller
def k8s_response(allowed, uid, message):
     return jsonify({"apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview", "response": {"allowed": allowed, "uid": uid, "status": {"message": message}}})

if __name__ == '__main__':
    warden.run(ssl_context=('certs/wardencrt.pem', 'certs/wardenkey.pem'),debug=True, host='0.0.0.0')Code language: PHP (php)

One of the requirements for an admission controller is that it is protected by certificates. So, let’s go create some certificates. I’ve created a script to generate these certificates for us and it’s stored in the git repository. The CN is important here, so it should match the DNS name of your admission controller.

NOTE: As of Kubernetes version 1.19 SAN certificates are required. If this will be deployed on 1.19 or higher, you must create a SAN Certificate. The github repository has been updated with an ext.cnf file and script will deploy a SAN certificate now.

Since I’ll be deploying this as a container within k8s, the service name exposing it is warden and the namespace it will be stored in will be validation. You should modify the script and the ext.cnf file before using it yourself.

keydir="certs"
cd "$keydir"

# CA root key
openssl genrsa -out ca.key 4096

#Create and sign the Root CA
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1024 -out ca.crt -subj "/CN=Warden Controller Webhook"

#Create certificate key
openssl genrsa -out warden.key 2048

#Create CSR
openssl req -new -sha256 \
    -key warden.key \
    -config ../ext.cnf \
    -out warden.csr

#Generate the certificate
openssl x509 -req -in warden.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out warden.crt -days 500 -sha256 -extfile ../ext.cnf -extensions req_ext

# Create .pem versions
cp warden.crt wardencrt.pem \
    | cp warden.key wardenkey.pemCode language: PHP (php)

The associated ext.cnf file is show below. Notice the alt_names section, which must match the name of your admission controller service. Also, feel free to update the country, locality, and organization to match your environment.

[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
prompt             = no
[ req_distinguished_name ]
countryName                 = US
stateOrProvinceName         = Illinois
localityName                = Chicago
organizationName            = HollowLabs
commonName                  = Warden Controller Webhook
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = warden.validation.svc

You’ll notice that the flask code is using these certificates and if you don’t change the names or locations in the script, it should just work. If you made modifications you will need to update the last line of the python code.

warden.run(ssl_context=('certs/wardencrt.pem', 'certs/wardenkey.pem'),debug=True, host='0.0.0.0')

Build and Deploy the Admission Controller

It’s time to build a container for this pod to run in. My Dockerfile is listed below as well as in the git repo.

FROM ubuntu:16.04

RUN apt-get update -y && \
    apt-get install -y python-pip python-dev

# We copy just the requirements.txt first to leverage Docker cache
COPY ./requirements.txt /app/requirements.txt

WORKDIR /app

RUN pip install -r requirements.txt

COPY . /app

ENTRYPOINT [ "python" ]

CMD [ "app/warden.py" ]Code language: PHP (php)

Push your image to your image registry in preparation for being deployed in your Kubernetes cluster.

Deploy the admission controller with the following Kubernetes manifest, after changing the name of your image.

---
apiVersion: v1
kind: Namespace
metadata:
   name: validation
---
apiVersion: v1
kind: Pod
metadata:
  name: warden
  labels:
    app: warden
  namespace: validation
spec:
  restartPolicy: OnFailure
  containers:
    - name: warden
      image: theithollow/warden:v1 #EXAMPLE- USE YOUR REPO
      imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: warden
  namespace: validation
spec:
  selector:
    app: warden
  ports:
  - port: 443
    targetPort: 5000
Code language: PHP (php)

Deploy the Webhook Configuration

The admission controller has been deployed and is waiting for some requests to come in. Now, we need to deploy the webhook configuration that tells the Kubernetes API to check with the admission controller that we just deployed.

The webhook configuration needs to know some information about what types of objects it’s going to make these REST calls for, as well as the URI to send them to. The clientConfig section contains info about where to make the API call. Within the rules section, you’ll define what apiGroups, resources, versions and operations will trigger the requests. This will seem similar to RBAC polices. Also note the failurePolicy which defines what happens to object requests if the admission controller is unreachable.

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validating-webhook
  namespace: validation
webhooks:
  - name: warden.validation.svc
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions: ["v1","v1beta1"]
    rules:
      - apiGroups: ["apps", ""]
        resources:
          - "deployments"
          - "pods"
        apiVersions:
          - "*"
        operations:
          - CREATE
    clientConfig:
      service:
        name: warden
        namespace: validation
        path: /validate/
      caBundle: #See command below 
    

The last piece of the config is the base64 encoded version of the CA certificate. If you used the script in the git repo, the command below will print the caBundle information for the manifest.

cat certs/ca.crt | base64

Deploy the webook. kubectl apply -f [manifest].yaml

Test the Results

There are three manifests in the /test-pods folder that can be used to test with.

test1.yaml – Should work with the admission controller because it has a proper billing label.

test2.yaml – Should fail, because there are no labels assigned.

test3.yaml – Should fail, because while it does have a label, it does not have a billing label.

Notice that each of these tests provided different responses. These responses can be customized so that you can give good feedback on why an operation was not permitted.