Using kubectl within the kubernetes cluster

TL;DR

The kubectl is used to run from outside of the kubernetes cluster.
This post will show how to use kubectl from within the cluster which can be run from a job, pod, deployment etc.

The solution includes the following requirements explained in the post:
1. Create a kubectl image with:
- Appropriate kubectl version
- User and UID with the permissions to use the kubeconfig on the host, by default located at ~/.kube/config.
- Provide the proper RBAC permissions to run the kubectl.

Note: The RBAC permissions may be of a security risk and should be handle with care.

This solution can be used for infinite scenarios.
The following is a demonstration of using a job running kubectl as an alternative to helm hooks (which caused in my scenario a race condition).
The solution solved the ability to wait for a resource without the race condition cause by the hook as explained bellow.

The Helm hooks issue

Helm hooks are mostly used to run a job / pod /deployment etc prior or after the installations. For example database migration prior to the pod upgrade or post installation database maintenance and the like. Of course there are many other use cases.

The Helm cli command would complete once all the resources are deployed, but before they are all available.

A post hook would not let helm to complete until all the resources are available to perform the post installation hook. This is fine and a normal behavior, unless you have a race condition conflict as it happened in my scenario described bellow.

My scenario

The following scenario is just an example of the implementation which eventually I found a way to solve with a hook, however I found the kubectl image solution very useful when there is a need to wait for a resource or perform any other action with kubectl from within the cluster.

In my case I had a kadalu storage implementation that creates the pv/pvc resources. The post installation hook created a race condition:
- The pod is pending for the pv/pvc to bond (waiting for helm to complete).
- The post hook waits for the pod to be available while it is stuck as pending.
- The post hook waits for all resources to be available and will not let helm to complete till then.
Hence a race condition.

The alternative - kubectl inside the cluster

Create a kubectl image:
Several kubectl images are available in GitHub and Docker Hub. I based my kubectl image from a simple Dockerfile which you can fine here:
My changes from the source:
- kubectl version
- User and UID same as the one running the helm / kubectl command on the host with permissions to use the kubeconfig file.

Dockerfile:

The kubectl was pushed to my private registry. You can push your Dockerfile to Docker Hub if needed.

Using a job with kubectl image to wait for my deployment has no race condition. The pod is created with the pv/pvc, the job uses kubectl to wait for the deployment to be available and once ready it runs the post installation. Voila!

The kubernetes Job

An initContainer is used to wait for the deployment.
A container is used for the post installation process.

post-install-job.yaml:

Ah one issue… You will get an RBAC error:

Error from server (Forbidden): deployments.apps "my-app-deployment" is forbidden: User "system:serviceaccount:your-ns:default" cannot list resource "deployments" in API group "apps" in the namespace "your-ns"

The RBAC permissions are required to provide kubectl to run within the cluster. A couple of manifests required: Role and RoleBinding:

kubectl-roles.yaml:

kubectl-rolebinding.yaml:

That’s it! Now the job waits in the initContainer for the deployment to available and ready and connected to the pv/pvc.
Once ready, a container is triggered to perform the post installation.

This was my scenario, there are unlimited use cases that can be solved with this solution.

Conclusion:

Creating a kubectl image and providing it with the proper RBAC permissions will allow you to run any kubectl command from within cluster.
My use case was to wait for a resource, however this solution opened a window to a whole world of infinite solutions to the day to day kubernetes and helm implementation.