Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
When using SUSE Rancher as Kubernetes deployment method (RKE), the Rancher "local" cluster runs the Rancher management API and UI. By default the incoming http/https connection is received by the Ingress (Nginx).
There are a couple of ways how to setup Rancher Kubernetes. You could deploy real certificates or use Let's Encrypt certificates directly on the Ingress service but in our setups we usually run a reverse proxy for additional security in front of any Kubernetes Ingress. We therefore keep the default certificate setting during the RKE run, which lets Rancher create and handle (internal) certificates.
As the communication between reverse proxy and Kubernetes nodes uses encrypted http (https), this means a TLS certificate is in use inside the Ingress pods running Nginx.
The internal certificates can easily be monitored by using a simple HTTPS connection (on default port tcp/443) to the Kubernetes nodes. The default TLS certificate will show up. With the monitoring plugins check_http (using -C flag) or check_ssl_cert, the certificate check can be added to your monitoring.
After a couple of months of running your (monitored) Kubernetes cluster, you might receive an alert, that the internal certificate (Kubernetes Ingress Controller Fake Certificate), used by the Nginx Ingress pods, is about to expire:
The certificates are created upon the first start of the Ingress Nginx pod and the same certificate is kept. There is no automatic renewal or certificate rotation (as we know it from the Kubernetes API certificates).
To renew the certificate about to expire, we first identify the Ingress Nginx pods running inside the ingress-nginx namespace:
ck@mint ~ $ kubectl -n ingress-nginx get pod
NAME READY STATUS RESTARTS AGE
default-http-backend-6977475d9b-zdjhl 1/1 Running 5 630d
nginx-ingress-controller-95gzb 1/1 Running 0 198d
nginx-ingress-controller-gv5j7 1/1 Running 5 630d
nginx-ingress-controller-pkqtm 1/1 Running 0 46d
To find out which pod runs on which node, use the -o wide flag:
ck@mint ~ $ kubectl -n ingress-nginx get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default-http-backend-6977475d9b-zdjhl 1/1 Running 5 630d 10.42.1.80 10.10.2.6 <none> <none>
nginx-ingress-controller-95gzb 1/1 Running 0 198d 10.10.1.127 10.10.1.127 <none> <none>
nginx-ingress-controller-gv5j7 1/1 Running 5 630d 10.10.2.6 10.10.2.6 <none> <none>
nginx-ingress-controller-pkqtm 1/1 Running 0 46d 10.10.1.208 10.10.1.208 <none> <none>
From the monitoring screenshot above, we know that the certificate is about to expire on the Ingress running on node onl-ran02-p, which has the internal IP 10.10.2.6.
Delete the pod in question (nginx-ingress-controller-gv5j7):
ck@mint ~ $ kubectl delete pod nginx-ingress-controller-gv5j7 -n ingress-nginx
Kubernetes will create a new pod, which can be seen after a few seconds:
ck@mint ~ $ kubectl -n ingress-nginx get pod
NAME READY STATUS RESTARTS AGE
default-http-backend-6977475d9b-zdjhl 1/1 Running 5 630d
nginx-ingress-controller-95gzb 1/1 Running 0 198d
nginx-ingress-controller-pkqtm 1/1 Running 0 46d
nginx-ingress-controller-wbdmt 0/1 Running 0 6s
By following the logs of this newly created pod, we can see that the TLS certificates are created from scratch (using default certificate):
ck@mint ~ $ kubectl logs nginx-ingress-controller-wbdmt -n ingress-nginx | egrep -i certificate
I0814 09:18:07.126070 8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
[...]
A few seconds later, monitoring turned green again:
We at Infiniroot love to share our troubleshooting knowledge when we need to tackle certain issues - but we also know this is not for everyone ("it just needs to work"). So if you are looking for a managed Kubernetes environment located in Switzerland or even in your own on-premise data center, check out our Private Kubernetes Container Cloud Infrastructure service at Infiniroot.