Lessons Learned while developing a Kubernetes Operator
Sergio Marcelino is a Full Stack Software Engineer at Entando working on the next generation of Entando’s cloud native, Kubernetes and microservices architecture. In this blog post he shares some lessons learned and tips about getting started with Kubernetes and operators in a development environment.
About one month ago, I started to work on a Kubernetes project at Entando and as we were planning how to do it, it was clear that we need to move forward with a Kubernetes Operator. So we did it.
One of my colleagues kickstarted the project and I got the task to orchestrate containers with Keycloak (a software to handle authentication and security stuff). We decided to let the operator to create a Keycloak instance on demand, then we needed to use it to create a client for the container with roles and permissions, then finally, provide the credentials out of the box to the container so it can use it to authenticate users and so on.
So my first challenge was to understand how to build all of that into the k8s operator, I started to develop the Keycloak instance creation. First of all I had to install a Kubernetes cluster on my machine, I chose Minikube.
My problem with Minikube
I have a very cool machine with 16GB RAM and 250 GB SSD m.2 which is 10x faster than a normal SSD. I've never had issues with memory until I started to work with Kubernetes and that was weird to me, because even though I have 16GB RAM, the SSD is so fast that in most of the times I use swap and never noticed the difference. But while trying to create the deployments the cluster was suddenly having a lot of problems.
Then I've done some research and realised that Kubernetes on MacOS does not run natively, it uses VirtualBox and the memory from my machine wasn't being used. By default it uses only 1GB, so you have to create the cluster passing the memory you want to use. First you have to delete the prior cluster in order to apply the changes.
Here some important commands.
How a pod can connect to another?
So I needed to create a MySQL pod and then create the Keycloak pod connecting to the database. Basically you will have to create a service exposing the port and then you can use the service name as the address.
My Keycloak pod was receiving a KILL Signal
Having successfully deployed the MySQL, then I created the Keycloak connecting to the database but I couldn't startup the Keycloak instance because the pod was receiving a KILL signal and I had no idea why.
After some research I saw one guy with a similar problem because his machine didn't have enough memory available, then I suspected that it would be some resources limiting on Kubernetes (it was also related to the Minikube issue). You have to put on deployment.
Conclusions and Outcome
Thinking about the first steps in Kubernetes, it is kinda difficult to memorize all those commands, so I think it's important to create a note with all of them and when to use.
Some errors and behaviours are hard to read and understand, I had to search on Google for answers on why one pod isn't being able to connect to another pod and things like that, although the Kubernetes documentation is extremely complete.
One thing that should be clear in the docs is the Minikube VirtualBox to allocate resources, I thought initially that my computer didn't have enough memory to handle that.