Skip to main content
Version: Next

Problem Statement

Zarf's agent-hook is responsible for mutating resources in the cluster to use the internal docker registry and git registry. The agent-hook uses a Kubernetes MutatingWebhookConfiguration to inject pull secrets into Kubernetes Pods, HelmReleases, and GitRepositories.

A MutatingWebhookConfiguration includes TLS certs for the Kubernetes API server to use when communicating with the pod responsible for performing the mutations. The TLS certs for Zarf's agent-hook pods have a year lifespan. As a result, clusters using Zarf will need to generate new TLS certs every year.

Failing to renew the TLS certs will result in pods failing to schedule in the cluster.

Example Error Log from Zarf agent-hook Pods​

When the TLS certs have expired, the agent-hook pods will log a similar error to the following:

error: tls: bad certificate
server.go:3214: http: TLS handshake error from 10.12.169.135:35492: remote error: tls: bad certificate

Verifying Expired Certs​

The TLS certs are stored in the zarf namespace and can be pulled out of the cluster and examined for expirary details.

If kubectl and openssl are installed, the following command can be used to examine the certificate to determine if it has expired:

kubectl get secret agent-hook-tls -n zarf --template='{{index .data "tls.crt" | base64decode}}' | openssl x509 -dates -noout

Example output:

notBefore=Jan  3 17:49:45 2025 GMT
notAfter=Jan 13 17:49:45 2026 GMT

Renewing Zarf agent-hook TLS Cert​

Zarf has a built-in tool for regenerating the TLS certs. The following command will generate new certs and reconfigure Zarf to use them:

zarf tools update-creds agent

Example output:

Agent TLS the certificates used to connect to Zarf's Agent

    Certificate Authority: **existing (sanitized)** -> **replacement (sanitized)**
    Public Certificate: **existing (sanitized)** -> **replacement (sanitized)**
    Private Key: **existing (sanitized)** -> **replacement (sanitized)**

? Continue with these changes? Yes
  ✔  Gathering information to update Zarf Agent TLS
  ✔  Updating values for helm release zarf-d2db14ef40305397791454e883b26fc94ad9615d
  ✔  Performing a rolling update for the Zarf Agent deployment

Optionally, the cert in the cluster can be examined to validate that it has been updated:

kubectl get secret agent-hook-tls -n zarf --template='{{index .data "tls.crt" | base64decode}}' | openssl x509 -dates -noout