Kubernetes on The IT Hollow

Configure a Private Registry for Tanzu Kubernetes Clusters

Wed, 22 Sep 2021 13:31:53 +0000

A really common task after deploying a Kubernetes cluster is to configure it to use a container registry where the container images are stored. A Tanzu Kubernetes Cluster (TKC) is no exception to this rule. vSphere 7 with Tanzu comes with an embedded harbor registry that can be used, but in many cases you all ready have your own container registry and so you’d like to continue using that instead.

vSphere 7 with Tanzu Updates

Thu, 13 May 2021 14:45:54 +0000

At some point, you’ll be faced with an upgrade request. New Kubernetes features, new security patches, or just to maintain your support. A vSphere 7 with Tanzu deployment has several components that may need to be updated and most of which can be updated independently of one another. In this post we’ll walk through an update to vSphere, then update the Supervisor namespace, and then finally the Tanzu Kubernetes cluster.

Customize vSphere 7 with Tanzu Guest Clusters

Mon, 01 Feb 2021 15:29:00 +0000

Kubernetes clusters can come in many shapes and sizes. Over the past 18 months I’ve deployed quite a few Kubernetes clusters for customers but these clusters all have different requirements. What image registry am I connecting to? Do we need to configure proxies? Will we need to install new certificates to the nodes? Do we need to tweak some containerd configurations? During many of my customer engagements the answer to the above questions is, “yes”.

Resizing Tanzu Kubernetes Grid Cluster Nodes

Wed, 09 Dec 2020 15:05:00 +0000

Have you ever missed when trying to properly size an Kubernetes environment? Maybe the requirements changed, maybe there were wrong assumptions, or maybe the project took off and it just needs more resources. Under normal circumstances, I might suggest to you to build a new Tanzu Kubernetes Grid (TKG) cluster and re-deploy your apps. Unfortunately, as much as I want to treat Kubernetes clusters as ephemeral, they can’t always be treated this way. If you need to resize your TKG nodes without re-deploying a new cluster, then keep reading.

Using YTT to Customize TKG Deployments

Mon, 09 Nov 2020 15:05:00 +0000

If you’ve worked with Kubernetes for very long, you’ve surely run into a need to manage YAML files. There are a bunch of options out there with their own benefits and drawbacks. One of these tools is called ytt and comes as part of the Carvel tools (formerly k14s).

If you’re working with the Tanzu Kubernetes Grid product from VMware, you’re likely to be using ytt to mange your TKG YAML manifests. This post aims to help you get started with using ytt for your own customizations.

Ingress Routing - TKG Clusters

Tue, 15 Sep 2020 14:05:00 +0000

If you have been following the series so far, you should have a TKG guest cluster in your lab now. The next step is to show how to deploy a simple application and access it through a web browser. This is a pretty trivial task for most Kubernetes operators, but its a good idea to know whats happening in NSX to make these applications available. We’ll walk through that in this post.

Deploying Tanzu Kubernetes Clusters on vSphere 7

Wed, 09 Sep 2020 14:15:00 +0000

This post will focus on deploying Tanzu Kubernetes Grid (TKG) clusters in your vSphere 7 with Tanzu environment. These TKG clusters are the individual Kubernetes clusters that can be shared with teams for their development purposes.

I know what you’re thinking. Didn’t we already create a Kubernetes cluster when we setup our Supervisor cluster? The short answer is yes. However the Supervisor cluster is a unique Kubernetes cluster that probably shouldn’t be used for normal workloads. We’ll discuss this in more detail in a follow-up post. For now, let’s focus on how to create them, and later we’ll discuss when to use them vs the Supervisor cluster.

Replace vSphere 7 with Tanzu Certificates

Mon, 31 Aug 2020 14:45:00 +0000

When setting up your vSphere 7 with Tanzu environment, its a good idea to update the default certificate shipped from VMware with your own certificate. This is a good security practice to ensure that your credentials are protected during logins, and nobody likes to see those pesky certificate warnings in their browsers anyway, am I right?

Create and Trust Certificate Authority

This section of the blog post is to create a root certificate. In many situations, you won’t need to do this since your organization probably already has a certificate authority that can be used to sign certificates as needed. Since I’m doing this in a lab, I’m going to create a root certificate and make sure my workstation trusts this cert first. After this, we can use the root certificate to sign our vSphere 7 certificates.

Connecting to a Supervisor Namespace

Mon, 24 Aug 2020 14:15:00 +0000

In this post we’ll finally connect to our Supervisor Cluster Namespace through the Kubernetes cli and run some commands for the first time.

In the last post we created a namespace within the Supervisor Cluster and assigned some resource allocations and permissions for our example development user. Now it’s time to access that namespace so that real work can be done using the platform.

First, login to vCenter again with the administrator@vsphere.local account and navigate to the namespace that was previously created. You should see a similar screen where we configured our permissions. In the Status tile, click one of the links to either open in a browser or copy the URL to then open in a browser.

Creating Supervisor Namespaces

Mon, 17 Aug 2020 14:15:00 +0000

Congratulations, you’ve deployed the Workload Management components for your vSphere 7 cluster. If you’ve been following along with the series so far, you’ll have left off with a workload management cluster created and ready to being configuring your cluster for use with Kubernetes.

The next step in the process is to create a namespace. Before we do that, it’s probably useful to recap what a namespace is used for.

Namespaces the Theory

Depending on your past experiences, a namespace will likely seem familiar to you in some fashion. If you have a kubernetes background, you’ll be familiar with namespaces as a way to set permissions for a group of users (or a project, etc) and for assigning resources. Alternatively, if you have a vSphere background, you’re used to using things like Resource Pools to set resource allocation.

vSphere 7 with Tanzu - Getting Started Guide

Tue, 14 Jul 2020 14:16:18 +0000

VMware released the new version of vSphere with functionality to build and manage Kubernetes clusters. This series details how to deploy, configure, and use a lab running vSphere 7 with Kubernetes enabled.

The instructions within this post are broken out into sections. vSphere 7 requires pre-requisites at the vSphere level as well as a full NSX-T deployment. Follow these steps in order to build your own vSphere 7 with Kubernetes lab and start using Kubernetes built right into vSphere.

Enable Workload Management

Tue, 14 Jul 2020 13:44:36 +0000

This post focuses on enabling the workload management components for vSphere 7 with Kubernetes. It is assumed that the vSphere environment is already in place and the NSX-T configuration has been deployed.

To enable workload management, login to your vCenter as the administrator@vsphere.local account. Then in the Menu, select Work

Within the Workload Management screen, click the ENABLE button.

The first screen in the wizard, will list your compatible vSphere clusters. These clusters must have HA and DRS enabled in fully automated mode. If you are missing clusters, make sure you have ESXi hosts on version 7 with HA and DRS enabled. You’ll also need a Distributed switch on version 7 for these clusters.

vSphere 7 with Kubernetes Environment and Prerequisites

Tue, 14 Jul 2020 13:42:33 +0000

This post describes the lab environment we’ll be working with to build our vSphere 7 with Kubernetes lab and additional prerequisites that you’ll need to be aware of before starting. This is not the only topology that would work for vSphere 7 with Kubernetes, but it is a robust homelab that would mimic many production deployments except for the HA features. For example, we’ll only install one (singular) NSX Manager for the lab where in a production environment would have three.

Tier-0 Gateway

Tue, 14 Jul 2020 13:39:41 +0000

This post will review the deployment and configuration of a Tier-0 gateway to provide north/south routing into the NSX-T overlay networks.

The Tier-0 (T0) gateway is where we’ll finally connect our new NSX-T backed overlay segments to the physical network through an NSX-T Edge which was previously deployed.

The Tier-0 gateway will connect directly to a physical VLAN and on the other side to our T1 router deployed in the previous post. From there, we should have all the plumbing we need to route to our hosts and begin using NSX-T to do some cooler stuff. In the end, the network topology will look something like this:

Tier-1 Gateway and NSX Segments

Tue, 14 Jul 2020 13:36:56 +0000

This post will focus on deploying our first NSX Gateway/Router and setting up our overlay segments. Before you can start these steps, the Edge nodes should be up and running so that they can support the Tier-1 gateways.

NSX uses two types of routers/gateways. We’ll start by using a Tier-1 (T1) router. These routers are usually used to pass traffic between NSX overlay segments. We could create NSX segments without any routers, but it would require a router to pass traffic between these segments so we will create a T1 router first.

Deploy NSX-T Edge Nodes

Tue, 14 Jul 2020 13:26:22 +0000

NSX-T Edge nodes are used for security and gateway services that can’t be run on the distributed routers in use by NSX-T. These edge nodes do things like North/South routing, load balancing, DHCP, VPN, NAT, etc. If you want to use Tier0 or Tier1 routers, you will need to have at least 1 edge node deployed. These edge nodes provide a place to run services like the Tier0 routes. When you first deploy an edge, its like an empty shell of a VM until these services are needed.

NSX Pools, Zones, and Nodes Setup

Tue, 14 Jul 2020 13:23:46 +0000

In the previous post we deployed an NSX Manager. Now it’s time to start configuring NSX so that we can build cool routes, firewall zones, segments, and all the other NSX goodies. And even if we don’t want to build some of these things, we’ll need this setup for vSphere 7 with Kubernetes.

Add an IP Pool

The first thing we’ll setup is an IP Pool. As you might guess, an IP Pool is just a group of IP Addresses that we can use for things. Specifically, we’ll use these IP Addresses to assign Tunnel Endpoints (Called TEPs previously called VTEPs in NSX-V parlance) to each of our ESXi hosts that are participating in the NSX Overlay networks. The TEP becomes the point in which encapsulation and decapsulation takes place on each of the ESXi hosts. Think of it this way, when encapsulated traffic needs to be routed to a VM on a host, what IP Address do we need to send the traffic to, so that it can reach that VM. This is the TEP. We need to setup a TEP on each host, and the IP Addresses for these TEPs come from an IP Pool. Since I have three hosts, and expect to deploy 1 edge nodes, I’ll need a TEP Pool with at least 4 IP Addresses. Size your environment appropriately.

NSX Installation

Tue, 14 Jul 2020 13:18:52 +0000

This post will focus on getting the NSX-T Manager deployed and minimally configured in the lab. NSX-T is a pre-requisite for configuring vSphere 7 with Kubernetes as of the time of this writing.

Deploy the NSX Manager

The first step in our build is to deploy the NSX Manager from an OVA template into our lab. The NSX Manager is the brains of the solution and what you’ll be interacting with as a user. Each time you configure a route, segment, firewall rule, etc., you’ll be communicating with the NSX Manager. Download and deploy the OVA into your vSphere lab.

Kubernetes Validating Admission Controllers

Tue, 26 May 2020 15:05:00 +0000

Hey! Who deployed this container in our shared Kubernetes cluster without putting resource limits on it? Why don’t we have any labels on these containers so we can report for charge back purposes? Who allowed this image to be used in our production cluster?

If any of the questions above sound familiar, its probably time to learn about Validating Admission Controllers.

Validating Admission Controllers - The Theory

Admission Controllers are used as a roadblocks before objects are deployed to a Kubernetes cluster. The examples from the section above are common rules that companies might want to enforce before objects get pushed into a production Kubernetes cluster. These admission controllers can be from custom code that you’ve written yourself, or a third party admission controller. A common open-source project that manages admission control rules is Open Policy Agent (OPA).

Kubernetes Liveness and Readiness Probes

Mon, 18 May 2020 14:00:00 +0000

Just because a container is in a running state, does not mean that the process running within that container is functional. We can use Kubernetes Readiness and Liveness probes to determine whether an application is ready to receive traffic or not.

Liveness and Readiness Probes - The Theory

On each node of a Kubernetes cluster there is a Kubelet running which manages the pods on that particular node. Its responsible for getting images pulled down to the node, reporting the node’s health, and restarting failed containers. But how does the Kubelet know if there is a failed container?

Kubernetes Pod Auto-scaling

Mon, 04 May 2020 14:05:00 +0000

You’ve built your Kubernetes cluster(s). You’ve built your apps in containers. You’ve architected your services so that losing a single instance doesn’t cause an outage. And you’re ready for cloud scale. You deploy your application and are waiting to sit back and “profit.”

When your application spins up and starts taking on load, you are able to change the number of replicas to handle the additional load, but what about the promises of cloud and scaling? Wouldn’t it be better to deploy the application and let the platform scale the application automatically?

Kubernetes Resource Requests and Limits

Mon, 20 Apr 2020 15:00:00 +0000

Containerizing applications and running them on Kubernetes doesn’t mean we can forget all about resource utilization. Our thought process may have changed because we can much more easily scale-out our application as demand increases, but many times we need to consider how our containers might fight with each other for resources. Resource Requests and Limits can be used to help stop the “noisy neighbor” problem in a Kubernetes Cluster.

Resource Requests and Limits - The Theory

Kubernetes uses the concept of a “Resource Request” and a “Resource Limit” when defining how many resources a container within a pod should receive. Lets look at each of these topics on their own, starting with resource requests.

In-tree vs Out-of-tree Kubernetes Cloud Providers

Tue, 14 Apr 2020 14:05:00 +0000

VMware offers a Kubernetes Cloud Provider that allows Kubernetes (k8s) administrators to manage parts of the vSphere infrastructure by interacting with the Kubernetes Control Plane. Why is this needed? Well, being able to spin up some new virtual disks and attaching them to your k8s cluster is especially useful when your pods need access to persistent storage for example.

The Cloud providers (AWS, vSphere, Azure, GCE) obviously differ between vendors. Each cloud provider has different functionality that might be exposed in some way to the Kubernetes control plane. For example, Amazon Web Services provides a load balancer that can be configured with k8s on demand if you are using the AWS provider, but vSphere does not (unless you’re using NSX).

Deploying Tanzu Kubernetes Grid Management Clusters - vSphere

Mon, 06 Apr 2020 14:00:00 +0000

VMware recently released the 1.0 release of Tanzu Kubernetes Grid (TKG) which aims at decreasing the difficulty of deploying conformant Kubernetes clusters across infrastructure. This post demonstrates how to use TKG to deploy a management cluster to vSphere.

If you’re not familiar with TKG yet, you might be curious about what a Management Cluster is. The management cluster is used to manage one to many workload clusters. The management cluster is used to spin up VMs on different cloud providers, and lay down the Kubernetes bits on those VMs, thus creating new clusters for applications to be build on top of. TKG is built upon the ClusterAPI project so this post pretty accurately describes the architecture that TKG uses.

Tanzu Mission Control - Access Policies