Overview
Given below are various commands that can be used to verify the health of the cluster during it's lifetime. These can be especially useful after an install or upgrade, which is when the chances of something not working as expected are higher.
Prerequisites
- Kubeconfig for target cluster
Validate All Composite Resource Definitions (XRDs)
A Crossplane XRD defines the schema for a custom API specification. XRDs create new API endpoints inside a Kubernetes cluster. After applying an XRD, Crossplane creates a new Kubernetes Custom Resource Definition (CRD) matching the defined API.
By verifying that the output from the commands below matches the expected output shown, we can ascertain that all XRDs have been established:
kubectl get xrd -ojson | jq '.items[] | select(.status.conditions[] | select(.type == "Established") | .status != "True") | .metadata.name'
Expected output:
<empty output>
And offered:
kubectl get xrd -ojson | jq '.items[] | select(.status.conditions[] | select(.type == "Offered") | .status != "True") | .metadata.name'
Expected output:
<empty output>
Validate All Crossplane Provider Kubernetes Objects
Provider Kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters typically provisioned by Crossplane. The Object resource type is used to manage Kubernetes Objects.
By verifying that the output from the commands below matches the expected output shown, we can ascertain that all Crossplane Provider Kubernetes Objects are ready:
kubectl get Object -ojson | jq '.items[] | select(.status.conditions[] | select(.type == "Ready") | .status != "True") | .metadata.name'
Expected output:
<empty output>
And synced:
kubectl get Object -ojson | jq '.items[] | select(.status.conditions[] | select(.type == "Synced") | .status != "True") | .metadata.name'
Expected output:
<empty output>
Validate All Claims
A Crossplane Claim represents a set of managed resources as a single Kubernetes object, inside a namespace. It is created by using the custom API defined in a Composite Resource Definition.
By verifying that the output from the commands below matches the expected output shown, we can ascertain that all Claims are ready:
kubectl get claim -A -ojson | jq '.items[].status.conditions[] | select( .type == "Ready" and .status != "True")' | grep -c status
Expected output:
0
And synced:
kubectl get claim -A -ojson | jq '.items[].status.conditions[] | select( .type == "Synced" and .status != "True")' | grep -c status
Expected output:
0
Validate All Pods And Containers
A Pod is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers.
By verifying that the output from the commands below matches the expected output shown, we can ascertain that all Pods are either running or have succeeded:
kubectl get pods -A -ojson | jq '.items[] | select( .status.phase != "Running" and .status.phase != "Succeeded" )' | grep -c phase
Expected output:
0
And no Containers in the Pods are stuck in the waiting state:
kubectl get pods -A -ojson | jq '.items[].status.containerStatuses[].state' | grep -c waiting
Expected output:
0
And no Containers in the Pods have been terminated due to an error:
kubectl get pods -A -ojson | jq '.items[].status.containerStatuses[].state.terminated | select( .reason != "Completed" )' | grep -c reason
Expected output:
0
Validate All Helm Releases (HRs)
A Helm Release is an instance of a Helm Chart running in a Kubernetes cluster. One chart can often be installed many times into the same cluster. And each time it is installed, a new Helm Release is created.
By verifying that the output from the commands below matches the expected output shown, we can ascertain that no HR are suspended:
kubectl get hr -A -ojson | jq '.items[]' | grep -c '"suspend": true'
Expected output:
0
And no HRs are rolled back or uninstalled:
kubectl get hr -A -ojson | jq '.items[].status.conditions[] | select( .type == "Remediated" and .status == "True")' | grep -c status
Expected output:
0
And all HRs are ready:
kubectl get hr -A -ojson | jq '.items[].status.conditions[] | select( .type == "Ready" and .status != "True")' | grep -c status
Expected output:
0
Validate All ReplicaSets
A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time.
By verifying that the output from the command below matches the expected output shown, we can ascertain that all ReplicaSets are available:
kubectl get rs -A -ojson | jq '.items[].status | select( .replicas != 0 and .replicas != .availableReplicas)' | grep -c replicas
Expected output:
0
Validate All Jobs
A Job manages the execution of one-off, batch tasks, ensuring they run to completion, and then stops.
By verifying that the output from the command below matches the expected output shown, we can ascertain that all Jobs have completed successfully:
kubectl get job --all-namespaces --field-selector status.successful=0 -o name
Expected output:
<empty output>
Validate All Crossplane Provider Keycloak Resources
Provider Keycloak is a Crossplane provider that enables managing Keycloak resources (like realms, clients, users, etc.) using Crossplane's infrastructure-as-code approach.
We can validate the following Keycloak resources created on the cluster.
Realm
A Keycloak Realm is a space where you manage objects, including users, applications, roles, and groups.
By verifying that the output from the commands below matches the expected output shown, we can ascertain that all Realms are ready and synced:
kubectl get Realm -ojson | jq '.items[].status.conditions[] | select( .type == "Ready" and .status != "True")' | grep -c status
kubectl get Realm -ojson | jq '.items[].status.conditions[] | select( .type == "Synced" and .status != "True")' | grep -c status
Expected output:
0
0
Group
A Keycloak Group provides a way to logically organize users, allowing you to apply roles and attributes to entire groups rather than individual users, simplifying user management and access control.
By verifying that the output from the commands below matches the expected output shown, we can ascertain that all Groups are ready and synced:
kubectl get Group -ojson | jq '.items[].status.conditions[] | select( .type == "Ready" and .status != "True")' | grep -c status
kubectl get Group -ojson | jq '.items[].status.conditions[] | select( .type == "Synced" and .status != "True")' | grep -c status
Expected output:
0
0
Client
A Keycloak Client is an entity that can request Keycloak to authenticate a user. Most often, clients are applications and services that want to use Keycloak to secure themselves and provide a single sign-on solution.
By verifying that the output from the commands below matches the expected output shown, we can ascertain that all Clients are ready and synced:
kubectl get Client -ojson | jq '.items[].status.conditions[] | select( .type == "Ready" and .status != "True")' | grep -c status
kubectl get Client -ojson | jq '.items[].status.conditions[] | select( .type == "Synced" and .status != "True")' | grep -c status
Expected output:
0
0