Enabling Vault

Configuring Vault Infrastructure as Code (IaC)

Vault IaC is disabled by default. The Vault IaC must be enabled and applied before enabling and deploying Vault in the SmoothGlue Zarf package.

Vault needs to use mutual SSL authentication. As a result, it must be configured to use passthrough ingress. Also, the Vault IaC module itself needs to be enabled.

Enabling the necessary IaC components and resulting Big Bang configuration is as simple as toggling these variables in the IaC inputs:

locals {
  cluster_inputs = {
    vault_passthrough_enable = true
  }

  modules = {
    vault = true
  }

  vault_inputs = {
    vault_kms_seal_enabled = true  # Enables AWS KMS key auto-unseal in Vault. If set to false, we
                                   # use the shamir key, which requires manual unseal.
  }
}

Initialize and Unseal Vault

When Vault is first installed in a new cluster, it will require some manual initialization, which will create some keys that need to be safely stored for later recovery. This initialization only needs to be done in one pod, vault-vault-0. Shell into this pod (choosing the vault container if given the option) and initialize Vault as shown in the following commands:

kubectl exec --stdin=true --tty=true vault-vault-0 -n vault  -- /bin/bash
vault operator init

This command could take up to a minute to complete, but it will give out five recovery keys and an Initial Root Token. Immediately copy the recovery keys and the Initial Root Token and save them in a safe space.

warning

Losing recovery keys can result in loss of data.

Then run:

vault login

This will prompt for the Initial Root Token. Paste it in (it won't show) and hit return, which should give a success message.

To check if this has initialized the other pods, run:

vault operator raft list-peers

This will ensure it returns all the vault pods.

Next run:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  issuer="https://kubernetes.default.svc.cluster.local"

This will enable Kubernetes, and they should separately give success notices.

Write a Vault policy for Prometheus with:

vault policy write prometheus-monitoring - << EOF
    path "/sys/metrics" {
      capabilities = ["read"]
    }
EOF

Then attach the policy to the existing monitoring-monitoring-kube-prometheus ServiceAccount with:

vault write auth/kubernetes/role/prometheus \
      bound_service_account_names=monitoring-monitoring-kube-prometheus \
      bound_service_account_namespaces=monitoring \
      policies="default,prometheus-monitoring" \
      ttl="15m"

Exiting this shell should show that all vault pods are up and running, as this initialization should automatically copy to the other vault pods. If there is still a Prometheus pod in the monitoring namespace that isn't running, delete it so it can come up again with the new settings.

If, for whatever reason, it is desired that Vault be reset/uninitialized, delete all of the PVCs for Vault, then delete the pods for vault-# and repeat the above instructions.