Kubernetes Validating Admission Controllers

Kubernetes Validating Admission Controllers

May 26, 2020 0 By Eric Shanks

Hey! Who deployed this container in our shared Kubernetes cluster without putting resource limits on it? Why don’t we have any labels on these containers so we can report for charge back purposes? Who allowed this image to be used in our production cluster?

If any of the questions above sound familiar, its probably time to learn about Validating Admission Controllers.

Validating Admission Controllers – The Theory

Admission Controllers are used as a roadblocks before objects are deployed to a Kubernetes cluster. The examples from the section above are common rules that companies might want to enforce before objects get pushed into a production Kubernetes cluster. These admission controllers can be from custom code that you’ve written yourself, or a third party admission controller. A common open-source project that manages admission control rules is Open Policy Agent (OPA).

A generalized request flow for new objects starts with Authenticating with the API, then being authorized, and then optionally hitting an admission controller, before finally being committed to the etcd store.

The great part about an admission controller is that you’ve got an opportunity to put our own custom logic in as a gate for API calls. This can be done in two forms.

ValidatingAdmissionController – This type of admission controller returns a binary result. Essentially, this means either yes the object is permitted or no the object is not permittetd.

MutatingAdmissionController – This type of admission controller has the option of modifying the API call and replacing it with a different version.

As an example, our example requirement that a label needs to be added to all pods, could be handled by either of these methods. A validating admission controller would allow or deny a pod from being deployed without the specified tag. Meanwhile, in a mutating admission controller, we could add a tag if one was missing, and then approve it. This post focuses on building a validating admission controller.

Once the API request hits our admission controller webhook, it will make a REST call to the admission controller. Next, the admission controller will apply some logic on the Request, and then make a response call back to the API server with the results.

The api request to the admission controller should look similar to the request below. This was lifted directly from the v1.18 Kubernetes documentation site.

The response REST call back to the API server should look similar to the response below. The uid must match the uid from the request in v1 of the admission.k8s.io api. The allowed field is true for permitting the request to go through and false if it should be denied.

Validating Admission Controllers – In Action

Now, let’s build our own custom validating admission controller in python using a Flask API. The goal of this controller is to ensure that all deployments and pods have a “Billing” label added. If you’d like to get a headstart with the code presented in this post, you may want to pull down the github repo: https://github.com/theITHollow/warden

We’ll assume that we’re doing some chargeback/showback to our customers and we need this label on everything or we can’t identify what customer it belongs to, and we can’t bill them.

Create the Admission Controller API

The first step we’ll go through is to build our flask API and put in our custom logic. If you look in the flask API example below, I’m handling an incoming request to the /validate URI and checking the metadata of the object for a “billing” label. I’m also capturing the uid of the request so I can pass that back in the response. Also, be sure to provide a message so that the person requesting the object gets feedback about why the operation was not permitted. Then depending on the label being found, a response is created with an allowed value of True or False.

This is a very simple example, but once you’ve set this up, use your own custom logic.

One of the requirements for an admission controller is that it is protected by certificates. So, lets go create some certificates. I’ve created a script to generate these certificates for us and its stored in the git repository. The CN is important here, so it should match the DNS name of your admission controller. Since I’ll be deploying this as a container within k8s, the service name exposing it is warden and the namespace it will be stored in will be validation. You should modify the script befor using it yourself.

You’ll notice that the flask code is using these certificates and if you don’t change the names or locations in the script, it should just work. If you made modifications you will need to update the last line of the python code.

warden.run(ssl_context=('certs/wardencrt.pem', 'certs/wardenkey.pem'),debug=True, host='')

Build and Deploy the Admission Controller

It’s time to build a container for this pod to run in. My Dockerfile is listed below as well as in the git repo.

Push your image to your image registry in preparation for being deployed in your Kubernetes cluster.

Deploy the admission controller with the following Kubernetes manifest, after changing the name of your image.

Deploy the Webhook Configuration

The admission controller has been deployed and is waiting for some requests to come in. Now, we need to deploy the webhook configuration that tells the Kubernetes API to check with the admission controller that we just deployed.

The webhook configuration needs to know some information about what types of objects it’s going to make these REST calls for, as well as the URI to send them to. The clientConfig section contains info about where to make the API call. Within the rules section, you’ll define what apiGroups, resources, versions and operations will trigger the requests. This will seem similar to RBAC polices. Also note the failurePolicy which defines what happens to object requests if the admission controller is unreachable.

The last piece of the config is the base64 encoded version of the CA certificate. If you used the script in the git repo, the command below will print the caBundle information for the manifest.

cat certs/ca.crt | base64

Deploy the webook. kubectl apply -f [manifest].yaml

Test the Results

There are three manifests in the /test-pods folder that can be used to test with.

test1.yaml – Should work with the admission controller because it has a proper billing label.

test2.yaml – Should fail, because there are no labels assigned.

test3.yaml – Should fail, because while it does have a label, it does not have a billing label.

Notice that each of these tests provided different responses. These responses can be customized so that you can give good feedback on why an operation was not permitted.