Configuration Instructions

The following are step-by-step instructions on how to configure SmoothGlue.

Keycloak

A SmoothGlue Build cluster comes with Keycloak. SmoothGlue will automatically configure the following within Keycloak:

• A smoothglue realm
• An admin group (_structsureAdmins)
• An audit group (_structsureAudit)
• Clients for the following applications
  • alertmanager
  • authservice
  • grafana
  • kiali
  • neuvector
  • prometheus

SmoothGlue will also configure the above set of applications to use their respective SSO client.

note

SmoothGlue does not currently configure applications or Keycloak clients for SmoothGlue Run clusters

For applications outside of the above list, an operator will need to create Keycloak clients and configure the application for SSO. Please refer to their respective SSO documentation (TODO: provide links)

There are a few options that need to be manually configured.

Logging in

The Admin username along with a randomly generated password can be retrieved from the cluster. They are stored in the keycloak-env Kubernetes secret in the keycloak namespace. Use these to log into Keycloak master realm by visiting https://keycloak.${DOMAIN}/auth/admin.

Configure SmoothGlue Admin Group

1. In the smoothglue realm, click on the top-level group named _structsureAdmins under the Groups option on the left column menu.
2. Inside that group, on the Role mapping tab, click Assign role.
3. Filter by clients, search for realm-admin, toggle the check box, and click the Assign button.

Optional: Configuring Keycloak Common Access Card (CAC)/Smart Card Authentication

Background Information

Keycloak supports logging in with an X.509 client certificate if you have configured the server to use mutual SSL authentication. As a result, Keycloak must be configured to use passthrough ingress.

Additionally, Keycloak must be provided with Certificate Authorities (CAs) to trust when verifying client certificates.

Within SmoothGlue, Keycloak can easily be configured to support CAC authentication. Some additional settings must be applied to the realm to enable users to login with their CAC.

Users will only be prompted for their CAC if they have it inserted before visiting the login page. If it is not present when visiting the login page, the user will be prompted for username and password or an error if username and password authentication has been disabled.

Configuring Keycloak for Passthrough Ingress

Enabling the necessary IaC components and resulting bigbang configuration is as simple as toggling this variable in the IaC inputs:

locals {
  cluster_inputs = {
    sso_passthrough_enable = true
  }
}

Enabling CAC Login

These steps are largely the same as Keycloak's documentation with some minor changes, given the format of CAC certificates.

1. While you are in the smoothglue realm, click on Authentication under Configure in the left pane.
2. Select the browser flow.
3. In the top right, select Action -> Duplicate.
   1. Name: Browser CAC Auth
   2. Description: Browser based authentication with CAC authentication
4. Click Duplicate.
5. Click Add step.
6. Search and select X509/Validate Username Form.
7. Click Add.
8. Drag X509/Validate Username Form block to be the second block just under Cookie.
9. In X509/Validate Username Form's Requirement drop-down, select Alternative.
10. Click on X509/Validate Username Form's settings (gear icon) and update the following:
    1. Alias: CAC Auth
    2. A regular expression to extract user identity: CN=(?:.*?)(\d*)(?:,|$)
    3. A name of user attribute: DOD_ID
    4. Check certificate validity: toggle on.
11. Click Save.
12. Click Action -> Bind flow:
    1. Choose binding type: Browser flow
13. Click Save.

Associating Users with CACs

Now that Keycloak can prompt users for their CAC, Keycloak needs to know what user is associated to which CAC.

For each user you want to associate:

1. While you are in the smoothglue realm, click on Users under Manage in the left pane.
2. Select the desired user.
3. Click the Attributes tab.
4. Add an attribute:
   1. Key: DOD_ID
   2. Value: The DOD ID number found on the back of their CAC.
5. Click Save.

Troubleshooting

Users Are Not Being Prompted for Their CAC

There are a few likely causes for this:

1. The Browser flow authentication flow, created in the steps above, is not bound for the smoothglue realm.
   1. Please ensure the authentication flow is bound so that users will be prompted for their CAC.
2. User has visited the login page before inserting their CAC.
   1. Have the user restart their browser or try visiting the login page from an incognito window.
3. Custom Keycloak configuration has overwritten settings necessary for CAC authentication.
   1. Please review the default configuration provided by SmoothGlue and Keycloak documentation to determine what was lost.
4. If you are providing a custom Truststore for validating smart cards in addition to CACs:
   1. Please ensure the custom Truststore also includes the DoD CAs required for CAC authentication.

X509 certificate authentication's failed. Invalid user

This error usually means the user attempted to login with their CAC, and Keycloak was unable to associate it to an account.

Please review their DoD ID and ensure it is set correctly within their user account attributes.

Jira

As Jira is used to manage users in Confluence, Jira needs to be set up before Confluence.

Instructions for setting up Jira

1. Go to https://jira.${DOMAIN}.
2. Follow Wizard to enter the license key.
3. Provide initial admin account information.
4. Skip the configuring email.
5. Go into the admin Applications tab and install Jira Software.
6. Set up the groups in Jira for syncing to Confluence: a. confluence-users b. confluence-administrators

Confluence

Instructions for setting up Confluence

1. Go to https://confluence.${DOMAIN}.
2. Go to the Confluence website provided.
3. Follow the Wizard to enter the license key.
4. Do the initial setup with Confluence-managed users and set your admin.
5. Under Users and security, select admin, and add a directory named Atlassian Jira. a. Input the server URL for Jira. b. Add read/write permissions.
6. Manager Users and Groups within Jira: a. Configure a system administrator account. b. Create a space name. Note: See Connecting Confluence to Jira applications for User Management here for instructions. c. Log in to Jira with the admin account. d. Go to Administration -> User Management -> Jira user server and click on Add application. e. Enter confluence for Application name. f. Enter a password that was generated for Password. g. Enter 0.0.0.0/0 for IP Addresses, and click Save. h. Log into Confluence with its admin account. i. Go to Administration > General Configuration > User directories. j. Add a directory and select type Atlassian Crowd. k. Enter Jira Server for Name. l. Enter the HTTP address for the server as the Server URL. m. Enter Confluence for Application Name. n. Enter the password that was defined for the Confluence application in the Jira settings. o. Enter 15 for Synchronization Interval to synchronize every 15 minutes. p. Click Test Settings, which should show the message "Connection test successful". q. Click Save and Test to finalize the configuration.
7. Log into Confluence with the smoothglue-console user and validate that it is an admin account.

Confluence's Collaborative Editing feature has intentionally been disabled in SmoothGlue due to auditing concerns. As a result, the confluence-synchrony pods have been disabled and Synchrony Connectivity health checks will show as failures in the Confluence UI. Some additional steps need to be taken after intial setup to completely disable Collaborative Editing. Please do the following when logged in as a Confluence administrator:

1. Click the gear icon in top right of Confluence UI.
2. Click General configuration.
3. In the left-hand panel, click Collaborative Editing.
4. Click Change mode.
5. Select Off.
6. Click Change.

note

Collaborative Editing may need to disabled twice before it shows it is disabled in the UI.

Mattermost

Mattermost needs to be set up prior to setting up Single Sign-On (SSO).

Instructions for setting up Mattermost

1. Go to https://chat.${DOMAIN}.
2. Register and sign in as the smoothglue-console user.
3. Create the Organization's "Welcome". Note: Everyone will have access to this, so avoid creating an Organization that is intended for use. a. Skip the rest of the Wizard set up.
4. In the System Console, enable access tokens: a. System Console -> Integrations -> Integration Management b. In the profile, create a PAT and save it to give to Console.
5. Keep this session open so that it can bestow permission on an SSO user.

Gitlab

Removing areas of concerns in Gitlab

The following includes areas of concern for the GitLab configuration.

• SSO Configuration
• Disabling Self-Registration
• Disabling Shared Gitlab-runner
• Disabling Public Projects
• Account Limitations
• Sign-up Restrictions
• External Authorization
• Repository Mirroring
• Spam/Anti-bot Protection
• Import and Export Rate limits
• Prevent Pushing Secret Files
• Prohibited File Names

Configuring Gitlab via the UI

The root (default admin username in GitLab) credentials can be retrieved from the cluster. The password is stored in the gitlab-gitlab-initial-root-password secret in the gitlab namespace.

Disabling Self-Registration

It is highly recommended to disable self-registration for GitLab. Ideally only users who have a user in Keycloak are allowed to login to GitLab.

Documentation from GitLab.

As an admin, select Menu -> Admin, under Settings -> General -> Sign-up restrictions. The settings for enabling/disabling Self Registration are located within.

Turn off Sign-up enabled.

Disable Shared Gitlab-runner

Documentation from GitLab.

As an admin on the left panel, click Admin, under the Settings -> CI/CD -> Continuous Integration and Deployment. The settings for Auto DevOps, shared runners, and artifacts are located within.

Turn the following off:

• Default to Auto DevOps pipeline for all projects.
• Enable shared runners for new projects.
• Enable pipeline suggestion banner.

Disable Public Project

Documentation from GitLab.

As an admin on the left panel, click Admin, under the Settings -> General -> Visibility and access controls. The settings for visibility are located within.

Set the following:

• Default project creation protection -> Maintainers
• Restricted visibility levels -> Public
• Note: If the public level is selected, it prevents outside users from seeing in and inside users from giving outside visibility.
• Enable Git Access protocols to Only HTTP(S)
• Disable Project export enabled.

Account Limitations

As an admin on the left panel, click Admin, under the Settings -> General -> Account and Limit. The settings for Account/Limits are located within.

Set the following:

• Uncheck User OAuth Applications.
• Uncheck Prompt users to upload SSH keys.
• Set Personal Access Token Prefix to 'smoothglue-'.
• Set Default project limits to 0.
• Check Deactivate dormant users after 90 days of inactivity.
• Uncheck the box for allow new users to create top-level groups.
• Uncheck the box for allow users to delete their own accounts.

External Authorization

Documentation from GitLab.

As an admin on the left panel, click Admin, under the Settings -> General -> External authorization. The settings for external authorizations are located within.

Repository Mirroring

Documentation from GitLab.

As an admin on the left panel, click Admin, under the Settings -> Repository -> Repository Mirroring. The setting for mirroring repositories is located within.

Set the following:

• Uncheck Allow Repository mirroring configuration.

Spam/Anti-bot Protection

As an admin on the left panel, click Admin, under the Settings -> Reporting -> Spam and Anti-bot Protection. The setting for Spam/Anti-bot protection is located within.

Set the following:

• Enable reCAPTCHA.
• Enable reCAPTCHA for login.
• Limit sign-in from multiple IP addresses.

Import and Export Rate limits

As an admin on the left panel, click Admin, under the Settings -> Network -> User and IP rate limits. The setting for Import/Export Rate limits is located within.

• Enable all Rate Limits

Prevent Pushing Secret Files

Documentation from GitLab.

As an admin, on the left sidebar, at the bottom, select Admin, select Push rules, expand Push rules. Set the rule as described in the link above to prevent committing secrets, such as credential files and SSH private keys to the repository. Select Save push rules.

Prohibited File Names

Documentation from GitLab.

As an admin, on the left sidebar, at the bottom, select Admin, select Push rules, expand Push rules. Set the rule as described in the link above to prevent committing prohibited file names to the repository. Select Save push rules.

All committed filenames cannot match this regular expression. If empty, any filename is allowed. This is to prevent accidental commits of sensitive files or malicious files.

note

Unfortunately there is a limit of 511 characters for this field.

Extension	Function	OS	Notes
.7z	Phishing File Archiver	Windows Mac Linux
.a3x	Executable Script	Windows
.appinstaller	Executable Double Click	Windows
.application	Phishing Executable Double Click	Windows
.appref-ms	Phishing Executable Double Click	Windows
.appx	Phishing Executable Double Click	Windows
.appxbundle	Phishing Executable Double Click	Windows
.cmd	Executable Script Double Click	Windows
.com	Executable Double Click	Windows
.cpl	Phishing Executable Double Click	Windows
.dll	Executable	Windows
.dmg	Executable Double Click	Mac
.docm	Phishing Double Click Macros	Windows Mac
.dotm	Phishing Double Click Macros	Windows Mac
.eml	Phishing	Windows Linux Mac
.exe	Executable Double Click	Windows
.hta	Executable Script Double Click	Windows
.lnk	Phishing Executable Double Click	Windows
.msc	Executable Double Click	Windows
.msi	Executable Double Click	Windows
.msix	Executable	Windows
.odt	Exploit Phishing	Windows
.pif	Executable Double Click	Windows
.potm	Phishing Double Click Macros	Windows Mac
.ppam	Phishing Double Click	Windows Mac
.ppsm	Phishing Double Click Macros	Windows Mac
.pptm	Phishing Double Click Macros	Windows Mac
.rar	Phishing File Archiver	Windows Mac Linux
.reg	Executable Script Double Click	Windows
.scr	Executable Double Click	Windows
.sldm	Phishing Double Click Macros	Windows Mac
.slk	Phishing Double Click Macros	Windows Mac
.theme	Phishing Double Click	Windows
.vbe	Executable Script Double Click	Windows
.vbs	Executable Script Double Click	Windows
.vhd	File Archiver	Windows
.vhdx	File Archiver	Windows
.wsf	Executable Script Double Click	Windows
.xlam	Phishing Double Click Macros	Windows Mac
.xlsb	Phishing Double Click Macros	Windows Mac
.xlsm	Phishing Double Click Macros	Windows Mac
.xpi	Package Management	Windows Mac Linux
.z	Phishing File Archiver	Windows Linux
.zip	Phishing File Archiver	Windows Mac Linux
.deb	Package Management	Debian-Based Distros
.rpm	Package Management	Windows Linux

Nexus Repository Manager

Instructions for setting up Nexus Repository Manager

Configuring Nexus Repository Manager Infrastructure as Code (IaC)

Nexus Repository Manager IaC is disabled by default and the RDS won't be created unless you pay for and enable the pro version of Nexus Repository Manager.

To Enable IaC set this variables in the IaC inputs:

locals {
  modules = {
    nexus = true
  }

}

If you purchased the Pro Version of Nexus Repository Manager create the RDS by setting this:

locals {
  nexus_inputs = {
    nexus_pro_version_enabled = true # If you have a license, you can enable this to turn on RDS for Nexus Repository Manager
  }
}

Vault

Instructions for setting up Vault

Configuring Vault Infrastructure as Code (IaC)

Vault IaC is disabled by default. It must be enabled and applied, as follows, before enabling and deploying Vault in the Zarf package.

Vault needs to use mutual SSL authentication. As a result, it must be configured to use passthrough ingress. Also, the Vault IaC module itself needs to be enabled.

Enabling the necessary IaC components and resulting Big Bang configuration is as simple as toggling these variables in the IaC inputs:

locals {
  cluster_inputs = {
    vault_passthrough_enable = true
  }

  modules = {
    vault = true
  }

  vault_inputs = {
    vault_kms_seal_enabled = true  # Enables AWS KMS key auto-unseal in Vault. If set to false, we
                                   # use the shamir key, which requires manual unseal.
  }
}

Initialize and Unseal Vault

When Vault is first installed in a new cluster, it will require some manual initialization, which will create some keys that need to be safely stored for later recovery. This initialization only needs to be done in one pod, vault-vault-0. Shell into this pod (choosing the vault container if given the option) and initialize Vault as shown in the following commands:

kubectl exec --stdin=true --tty=true vault-vault-0 -n vault  -- /bin/bash
vault operator init

This command could take up to a minute to complete, but it will give out five recovery keys and an Initial Root Token. Immediately copy the recovery keys and the Initial Root Token and save them in a safe space.

warning

Losing recovery keys can result in loss of data.

Then run:

vault login

This will prompt for the Initial Root Token. Paste it in (it won't show) and hit return, which should give a success message.

To check if this has initialized the other pods, run:

vault operator raft list-peers

This will ensure it returns all the vault pods.

Next run:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  issuer="https://kubernetes.default.svc.cluster.local"

This will enable Kubernetes, and they should separately give success notices.

Write a Vault policy for Prometheus with:

vault policy write prometheus-monitoring - << EOF
    path "/sys/metrics" {
      capabilities = ["read"]
    }
EOF

Then attach the policy to the existing monitoring-monitoring-kube-prometheus ServiceAccount with:

vault write auth/kubernetes/role/prometheus \
      bound_service_account_names=monitoring-monitoring-kube-prometheus \
      bound_service_account_namespaces=monitoring \
      policies="default,prometheus-monitoring" \
      ttl="15m"

Exiting this shell should show that all vault pods are up and running, as this initialization should automatically copy to the other vault pods. If there is still a Prometheus pod in the monitoring namespace that isn't running, delete it so it can come up again with the new settings.

If, for whatever reason, it is desired that Vault be reset/uninitialized, delete all of the PVCs for Vault, then delete the pods for vault-# and repeat the above instructions.

Scale Up Horizontal Pod Autoscaling (HPA)

Jira and Confluence require that only one pod be running during their initial setup. Afterwards, Jira and Confluence need to be configured for High Availability by scaling their statefulset / HPA.

Jira and Confluence

Jira

The maximum replica count for Jira's HPA is already configured to allow up to three pods. The Big Bang values included as an output from the IaC code provide a commented out value for setting the minimum replica count. Simply uncomment the following:

packages:
  jira:
    values:
      # replicaCount: 2

• Verify the health checks by going to the top right Settings gear -> General configuration -> Left panel -> Troubleshooting and support tools.
• Verify clustering by going to Clustering.

Confluence

The maximum replica count for Confluence's HPA is already configured to allow up to three pods. The Big Bang values included as an output from the IaC code provide a commented out value for setting the minimum replica count. Simply uncomment the following:

packages:
  confluence:
    values:
      # replicaCount: 2

• Verify the Confluence pod replication on the cluster by running kubectl get all -n confluence. This should show that the horizontalpodautoscaler has a MINPODS of 2 and a MAXPODS of 3.
• Verify the health checks by going to the top right Settings gear -> General configuration -> Left panel -> Troubleshooting and support tools.
• Verify clustering by going to Clustering.

Initialize Console

To initialize Console, the following clients need to be set up to connect to Console.

Jira, Confluence, GitLab, and Mattermost

Jira/Confluence

If Confluence is set up to use Jira's user database, then Jira and Confluence share the same users. The following addresses these console environment variables:

• JIRA_USERNAME
• JIRA_PERSONAL_ACCESS_TOKEN
• CONFLUENCE_PERSONAL_ACCESS_TOKEN

Create a Jira/Confluence user:

1. Log in to Jira with an admin-level account.
2. Go to User Management in the admin panel.
3. Generate a strong password: openssl rand -base64 64 | tr -d '\n' | pbcopy.
4. Create user as: a. Email address: Use an email address designated for Console. b. Full name: console. c. Username: console. d. Password: Use generated password.
5. Edit user groups for console to add: a. jira-administrators b. confluence-users c. confluence-administrators

Create Personal Access Token:

1. Log in as console.
2. Click the user icon in the top right corner, and go to Profile.
3. Create Personal Access Token.
4. Save token.

Repeat the Create Personal Access Token steps for Confluence.

GitLab

The following addresses use these console environment variables:

• GITLAB_PERSONAL_ACCESS_TOKEN

Create GitLab user:

1. Log into GitLab with an admin-level account.
2. Go to Users in the admin panel.
3. Create user as: a. Name: console. b. Username: console. c. Email: Use an email address designated for Console. d. Access Level: Administrator.
4. Generate a strong password: openssl rand -base64 64 | tr -d '\n' | pbcopy.
5. Manually set the password on the account.

Create Personal Access Token:

1. Impersonate or log in as console.
2. Click the user icon in the top right corner, and select Preferences.
3. Click Access Tokens on the left-hand panel.
4. Create a Personal Access Token with the api and the admin_mode scope.
5. Save token.

Mattermost

The following addresses use these console environment variables:

• MATTERMOST_ACCESS_TOKEN
• MATTERMOST_PERSONAL_ACCESS_TOKEN

Currently, the above variables are the same token.

Assign the PAT (saved when Mattermost was set up) to both MATTERMOST_ACCESS_TOKEN and MATTERMOST_PERSONAL_ACCESS_TOKEN.