An interesting challenge came up an work recently about how to do TLS Client Authentication from Kubernetes pods with short lived certificates.
The main gist was how to dynamically update the certificates on a short time frame (e.g. once a week) but without needing to manually mess about with recreating the Secrets that hold the keys and certificates
Cert-Manager
The standard tool for dealing with Certificate Authorities in Kubernetes is cert-manager, the usual use case is to issue server certificates either linked to an Ingress objects by adding annotations to these objects. These can then be provisioned from the CA, usually by using ACME. I set up a Small Step CA with ACME to issue certificates on my LAN.
This basically works by creating a Kubernetes Web Hook that looks for when Ingress are created or modified and then reading the annotation and creating a Certificate Signing Request which is then submitted to the CA for signing and the resultant Certificate and Key are stored in a Kubernetes Secret.
Cert-Manager-CSI
This allows for the creation of Certificates for things that are not exposed by Ingress e.g. where the TLS termination is being done in the Pod rather than be delegated to the HTTP reverse proxy providing the Ingress service or for Client Certificates.
The CSI driver allows virtual volumes to be created that mount the Secret that holds a x509 Certificate and Private Key to be mounted as files into a Pod.
When I first installed the CSI driver I couldn’t get it to work as it kept complaining that /var/snap was a Read Only file system, this turned out to be because I use microk8s as my Kubernetes distribution and this has a none standard kubelet directory location because it is packaged as a snap bundle. The solution is to pass the correct kubelet path to the helm chart
$ helm upgrade cert-manager-csi-driver oci://quay.io/jetstack/charts/cert-manager-csi-driver \ --install \ --namespace cert-manager \ --set app.kubeletRootDir=/var/snap/microk8s/common/var/lib/kubelet \ --wait
Small Step Certificate Issuer
As I already have a Small Step CA up and running to provide server certificates via ACME I needed to add a new provisioner that would create client certificates with the correct flags. I could probably have reused to the one I set up to do email signing but I decided to keep them separate for this test.
Details of how to install and configure the required tools can be found here.
$ step ca provisioner add "step-client-certs" --type JWK --create \ --x509-min-dur 24h --x509-max-dur 720h --x509-default-dur 720h
Keep a note of the password created as part of this process as it will need to be added to a Secret later, the kid of the provisioner is also required.
$ kubectl create secret -n step-issuer generic step-client-certs-password --from-literal=password=TopSecret
The Step Issuer is installed with
$ helm install \ step-issuer smallstep/step-issuer \ --namespace step-issuer \ --create-namespace
Then either a StepIssuer or a StepClusterIssuer I’m going to user a Cluster scoped Issuer so I don’t have to create one in every namespace I’m using.
apiVersion: certmanager.step.sm/v1beta1
kind: StepClusterIssuer
metadata:
name: step-client-issuer
namespace: step-issuer
spec:
url: https://certs.loc:9000
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJwekNDQVU2Z0F3SUJBZ0lSQUp1MnByWWJFa09HRGRBSnpORXErYWt3Q2dZSUtvWkl6ajBFQXdJd01qRVQKTUJFR0ExVUVDaE1LU0dGeVpHbHNiQ0JEUVRFYk1Ca0dBMVVFQXhNU1NHRnlaR2xzYkNCRFFTQlNiMjkwSUVOQgpNQjRYRFRJeU1USXdOREUxTWpVME1sb1hEVE15TVRJd01URTFNalUwTWxvd01qRVRNQkVHQTFVRUNoTUtTR0Z5ClpHbHNiQ0JEUVRFYk1Ca0dBMVVFQXhNU1NHRnlaR2xzYkNCRFFTQlNiMjkwSUVOQk1Ga3dFd1lIS29aSXpqMEMKQVFZSUtvWkl6ajBEQVFjRFFnQUVPRXI2ZFdZL1JjdStHNzBUV0FVbWVSVHQycFg1cEpwTy9YNkJHSHM3NkdlaQpWcFhGeTZ1YXFFeHVCc2xIdGRiMGtUanNkbWlnM2l5ckhFQmhXRVFCd3FORk1FTXdEZ1lEVlIwUEFRSC9CQVFECkFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUV3SFFZRFZSME9CQllFRkhSOG1nUndMUGN0UkZmV3B6Rk0KMk5mMDFkT2RNQW9HQ0NxR1NNNDlCQU1DQTBjQU1FUUNJSDc1bUh2cUxsRDRmYy94ZmFCZjVpc2htRWVJZW4xQQpiVEU5VTZxODRnZHdBaUJUVEZaWjB1dzJBSlpmQ1ZuWmFSQ3hmYnhzYURZNFBSQTNaMjdTczBkd2FRPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
provisioner:
name: step-client-certs
kid: IdYupAKmyDlfKPQlDP5IAqVhzU3eC0Au0OUj1wxDeTY
passwordRef:
name: step-client-certs-password
key: password
namespace: step-issuer
Testing it all
I have a standard bare Ubuntu pod I tend to use for debugging when I need a shell in the cluster. I modified it to request a client certificate and mount the files on /tls
apiVersion: v1
kind: Pod
metadata:
name: ubuntu
labels:
app: ubuntu
spec:
containers:
- image: "ubuntu:24.04"
command:
- "tail"
- "-f"
- "/dev/null"
imagePullPolicy: IfNotPresent
name: ubuntu
volumeMounts:
- name: tls
mountPath: /tls
volumes:
- name: tls
csi:
driver: csi.cert-manager.io
readOnly: true
volumeAttributes:
csi.cert-manager.io/issuer-name: step-client-issuer
csi.cert-manager.io/issuer-group: certmanager.step.sm
csi.cert-manager.io/issuer-kind: StepClusterIssuer
csi.cert-manager.io/common-name: ${POD_NAME}-${POD_NAMESPACE}
csi.cert-manager.io/key-usages: digital signature,client auth
csi.cert-manager.io/duration: 720h
restartPolicy: Always
The important configuration parts as the volumeAttributes which set which Issuer to use and what Common Name to set.
$ kubectl exec -it ubuntu -- /bin/bash root@ubuntu:/# cd /tls root@ubuntu:/tls# ls ca.crt tls.crt tls.key root@ubuntu:/tls# cat tls.crt -----BEGIN CERTIFICATE----- MIIDADCCAqegAwIBAgIQScv/Sg70g2nvMynmAej51jAKBggqhkjOPQQDAjA6MRMw EQYDVQQKEwpIYXJkaWxsIENBMSMwIQYDVQQDExpIYXJkaWxsIENBIEludGVybWVk ...