In this blog post, I will show you how to write a series of end to end tests which can validate the Kubernetes cluster health and provide immediate feedback. These tests provides a good level of abstraction and will run fine irrespective of where you are deploying your Kubernetes cluster — Azure or AWS.

Recently I have been working on a Python Library to orchestrate the deployment of MicroServices to a Kubernetes/Istio Service Mesh hosted in Cloud.
As part of this effort, we wanted to have an end to end testing strategy against our Kubernetes/Istio cluster and validate that the system is functioning as expected.

We have been writing unit tests for all our python code, but in a distributed system like Kubernetes there is every chance that a code change may pass all unit tests, but create unexpected issues at the system level. This is where having a solid End to End Testing strategy saves the day.

As the first step, we wanted to have a set of reliable end to end tests, which we can execute against our Kubernetes cluster and determine if the K8s cluster is in a good state.
Before we start with our deployment, we wanted to determine the overall health of the K8s cluster –

• We want to check if all the pods are running
• We want to detect the existence of any Crash Loops
• We want to look at the health of the various Master Components (Scheduler, Controller and Cluster Store – etcd) and Nodes (Kubelet, Kube-Proxy, Pods )

Now let’s move to the code and I will show you the E2E tests I have written. I am using the Python API Client for Kubernetes to interact with the cluster in Cloud. I am primarily using the CoreV1Api class and the method it provides for the tests below.

The first step is to setup the client for the End to End Tests. The Authentication and Kubernetes cluster information are stored in the kube-config file.

The below test iterates through the K8 Master Components and verifies if they are Healthy —

The below test verifies if the status of the Master Controller is Healthy —

The below test iterates through all the nodes and verifies if they are Ready. Incase there are any issues with the Kubelet it will also identify the exact issue —

The below test iterates through all the pods in the ‘default’ namespace and verifies if they are in a Running state —

Official Python client library for Kubernetes –
https://github.com/kubernetes-client/python

Documentation for Kubernetes API Endpoints –
https://github.com/kubernetes-client/python/tree/master/kubernetes#documentation-for-api-endpoints

In my next blogpost, I will show you how to write additional tests to verify other objects in the K8s cluster like pods, services and namespaces. Stay Tuned!

Also please let me know if you have any questions and I would be happy to discuss.

Categories: Kubernetes