Skip to main content
Version: 6.13.0

Problem Statement

It is good security practice to periodically rotate the database (Amazon Aurora and RDS) passwords for each application that uses them. This guide demonstrates how to manually rotate these passwords. We will likely incorporate automated password rotation in the future, but for now, the following include manual steps for doing this.

note

This guide only deals with rotating the RDS/Aurora database password used by the application to connect to the database, and not the master RDS/Aurora password.

Prerequisites​

  • Kubeconfig for target cluster

Required Tools​

  • Terraform (for IaC)
  • Terragrunt (for IaC)
  • Zarf

Overview​

It is prudent to scale down all applications using RDS and Aurora to 0 pods before changing the database passwords in order to avoid possible database corruption issues in the midst of a password change. The following are the steps to change the application database passwords.

  1. Scale down the applications.
  2. Taint the RDS/Aurora passwords.
  3. Run terragrunt apply.
  4. Re-deploy the Zarf package.
  5. Scale up the applications.
  6. Log back into the applications to verify functionality.

The applications currently using RDS and Aurora databases include:

  1. Confluence (Aurora)
  2. Jira (Aurora)
  3. Mattermost (Aurora)
  4. Keycloak (Aurora)
  5. Nexus Repository Manager (Aurora, shared with Nexus IQ)
  6. Nexus IQ (Aurora, shared with Nexus Repository Manager)
  7. SonarQube (Aurora)
  8. GitLab (RDS)
  9. Console (Aurora)

Scale Down the Applications​

This can be done using the commands shown below for each application.

kubectl scale statefulsets -n confluence confluence --replicas=0

kubectl scale statefulsets -n jira jira --replicas=0

kubectl scale deployment,rs --all -n mattermost-operator  --replicas=0
kubectl scale deployment,rs --all -n mattermost --replicas=0

kubectl scale statefulsets -n keycloak keycloak --replicas=0

kubectl scale deployment -n nexus-repository-manager nexus-repository-manager --replicas=0
kubectl scale deployment -n nexusiq nexus-iq --replicas=0

kubectl scale statefulsets -n sonarqube sonarqube-sonarqube --replicas=0

kubectl scale deployment -n gitlab gitlab-webservice-default gitlab-toolbox gitlab-registry gitlab-gitlab-shell  gitlab-gitlab-exporter gitlab-sidekiq-all-in-1-v2   --replicas=0
kubectl scale statefulsets -n gitlab gitlab-redis-master gitlab-gitaly --replicas=0

kubectl scale deployment -n console console --replicas=0

Taint the RDS/Aurora Password​

Taint the RDS/Aurora database password so that it is forcibly replaced by Terraform. Before tainting, first run terragrunt init through the same process (using the same values) as when initially created.

terragrunt --terragrunt-working-dir modules/confluence taint "random_password.this"

terragrunt --terragrunt-working-dir modules/jira/ taint "random_password.this"

terragrunt --terragrunt-working-dir modules/mattermost taint "random_password.this"

terragrunt --terragrunt-working-dir modules/keycloak taint "random_password.this"

terragrunt --terragrunt-working-dir modules/nexus taint "random_password.this"
terragrunt --terragrunt-working-dir modules/nexus taint "random_password.iq_password"

terragrunt --terragrunt-working-dir modules/sonarqube taint "random_password.this"

terragrunt --terragrunt-working-dir modules/gitlab taint "random_password.this"

terragrunt --terragrunt-working-dir modules/console taint "random_password.database_password_suffix"

Run terragrunt apply​

Once the database passwords have been tainted, run terragrunt apply to replace the passwords and apply the change. To do this, run terragrunt apply through the same process (using the same values) as when initially created.

Re-deploy the Zarf Package​

Now that the IaC has been updated and applied, run zarf package deploy to propagate the new database passwords into the cluster configuration (application secrets). To do this, run zarf package deploy through the same process (using the same values) as when last updated/created.

Scale up the Applications​

Most of the applications should scale back up after re-deploying the Zarf package. But some applications may stay scaled down; they can be scaled up using the same commands as used for scaling them down, but with --replicas=1.

Log Back into the Applications to Verify Functionality​

Log back into each application to verify functionality. Typically, if something went wrong (for example, if there is some password mis-match), then the application pod itself will fail to come up and the pod logs should indicate the error.