Kaptain’s default Certificate Authority or CA, which is handled by the Kommander component of DKP, creates and renews certificates automatically and periodically. A CronJob enabled by default verifies if a new certificate has been issued since the last run, and if so, updates the Secret with the new CA bundle, and restarts the authservice pod to make the new certificate effective.

You can establish the regularity of these jobs or manually trigger the reconciliation of your system with a renewed certificate to avoid or minimize downtime.

Enable or disable automatic certificate verification

Enable or disable a CronJob that periodically scans for any newly issued certificate.

The ingress.enableSecretRotation field of the ConfigMap containing Kaptain’s configuration is set to true by default, allowing the CronJob to run automatically:

ingress:
    enableSecretRotation: true
    secretRotationSchedule: "0 0 * * *"
CODE

To disable the job, set the enableSecretRotation field to false.

Change the regularity of the CronJob

You can modify the regularity of the CronJob to suit your needs. A CronJob that runs more regularly consumes larger amounts of resources, but minimizes possible downtime. Less regular CronJobs save up resources but increase the possibility of experiencing downtime. The default CronJob is set to run every day at 12 am.

The reconciliation is triggered by the CronJob and not by the certificate creation. That is why you may experience downtime when a new certificate has been issued, and the CronJob has not fetched the new value yet.

To change the regularity of the jobs, modify the secretRotationSchedule of the Kaptain ingress, following the standards for Cron scheduling expressions. In the following example, the schedule has been set to every 6 hours:

ingress:
    enableSecretRotation: true
    secretRotationSchedule: "* */6 * * *"
CODE

Trigger certificate reconciliation manually

You can also run the CronJob on-demand. This is useful when, for example, you know the Secret has been updated with a new certificate, but you don’t want to wait for the next scheduled run to restart the pods. If your cluster is experiencing downtime due to expired certificates, this command will make your cluster available again:

kubectl -n kaptain-ingress create job --from=cronjob/kaptain-ingress-cert-secret-rotation rotation-1
CODE

Verification

You can verify that the CronJob or manually triggered job ran successfully by verifying the kubeflow-oidc-ca-bundle secret and the authservice pod.

Verify the secret

Verify that the kubeflow-oidc-ca-bundle secret has been updated:

kubectl get secret kubeflow-oidc-ca-bundle -n kaptain-ingress
CODE

The output should look similar to this:

NAME                      TYPE     DATA   AGE
kubeflow-oidc-ca-bundle   Opaque   1      109m
CODE

If the secret has been recreated successfully, the value under AGE will be recent. 

Verify the pods

Verify if the authservice pod has been restarted:

kubectl get pod authservice-0 -n kaptain-ingress
CODE

The output should look like this:

NAME                              READY   STATUS    RESTARTS   AGE
authservice-0.                    1/1     Running   0          128m
CODE

If the pod has been restarted successfully, the value under AGE will be recent.