In-tree vs Out-of-tree Kubernetes Cloud Providers

VMware offers a Kubernetes Cloud Provider that allows Kubernetes (k8s) administrators to manage parts of the vSphere infrastructure by interacting with the Kubernetes Control Plane. Why is this needed? Well, being able to spin up some new virtual disks and attaching them to your k8s cluster is especially useful when your pods need access to persistent storage for example.

The Cloud providers (AWS, vSphere, Azure, GCE) obviously differ between vendors. Each cloud provider has different functionality that might be exposed in some way to the Kubernetes control plane. For example, Amazon Web Services provides a load balancer that can be configured with k8s on demand if you are using the AWS provider, but vSphere does not (unless you’re using NSX).

Originally, these cloud providers were maintained by the same folks writing the Kubernetes software. Given the number of cloud providers that might want to add functionality for their own platform, this wasn’t a very sustainable way to provide access to new cloud features. To fix this issue, and allow the Kubernetes project to move forward without the cruft of managing the providers, the Cloud Provider Interface was used to allow third parties to write their own cloud provider tools to enhance Kubernetes without needing access to the core Kubernetes project.

The main thing that I hope users realize, is that when you setup your cluster, you can currently pick between two types of cloud providers:

• In-tree cloud provider
• Out-of-tree (External) Cloud Provider

If you’ve followed my Kubernetes posts, you may notice that I’ve been using the in-tree cloud providers up until this point. Going forward, I’ll be using the external cloud providers when possible which should have better functionality than the in-tree providers, or will soon.

Why am I telling you this? Well even if we ignore the fact that external cloud providers may have a very different architecture including their own controllers, they may have different configuration specs as well. Lets take a look at the vSphere Storage Integration that comes with the In-tree cloud provider vs the External Cloud Provider.

StorageClass Differences – Example

In this post on Cloud Providers and Storage Classes, I used the in-tree cloud provider for vSphere and created a storage class from the yaml manifest below.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: vsphere-disk #name of the default storage class
  annotations:
    storageclass.kubernetes.io/is-default-class: "true" 
provisioner: kubernetes.io/vsphere-volume
parameters:
  diskformat: thin

Recently, I’ve been deploying my clusters in my lab through VMware Tanzu Kubernetes Grid (TKG) like in this post. TKG does not use the in-tree cloud provider for vSphere. To deploy a storage class (with similar functionality to above) on TKG the yaml manifest looks like this:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: vsphere-disk
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "k8s"

Now at first glance those manifests look very similar, but the “provisioner” is different. Its a different mechanism providing storage, so the provisioner is also different. Once you understand that, then the parameters section should also make sense about why they are different. In the in-tree storageclass I set a disk format. In the Out-of-tree cloud provider I have a VMware Storage Policy configured in vSphere and I reference that instead. Expect that your parameters and configs will be different for each provider and be sure to check if you’re using in-tree vs external providers.

I would make this additional note. Both of the storage classes can be deployed to my cluster at the same time. The in-tree vSphere cloud provider is still a valid Kubernetes manifest and can be applied, but if your Cloud Provider isn’t setup correctly it will fail to provision PVCs when you request them. You can see below that I’ve deployed a StorageClass and a PVC successfully, but my PVC wouldn’t bind to the storage and the message returned is a “Failed to provision volume with StorageClass “name”: Cloud Provider not installed properly” message.

Summary

The long and the short of this post is to make sure users understand that there are two different types of cloud providers used in Kubernetes. The in-tree provider, and the external providers such as VMware’s Cloud Provider (VCP). They have similar functionality, but they are different, which means your configurations might need to change based on which cloud provider you’re using.

For reference here are links for additional reading:

• Kubernetes In-Tree Cloud Provider Info
• vSphere External Cloud Provider

I wanted to write up a quick article here in case anyone getting started with Kubernetes gets tripped up by this as I did. I’ve had this tweet pinned to my timeline for a bit now and I’ve got to live up to that. I hope this helps clear up any confusion.

“Pass on what you have learned. Strength, mastery… but weakness, folly, failure also. Yes failure most of all. The greatest teacher, failure is.” – Yodahttps://t.co/R7RXvmoLGm

— Eric Shanks (@eric_shanks) March 25, 2020