Skip to main content
Version: 6.16.1

Release Notes

6.16.1 (2025-07-17)​

In addition to the features included in v6.16.0, this patch release also includes the following:

🪲 Bug Fixes​

  • IaC: The offline Terraform bundle for the ALB IaC module now references a cached local version of the load balancer module, in order to avoid queries to the upstream Terraform registry.
  • IaC: The eks-cluster variable commit_ref_name has been removed and resources such as EC2 instances are no longer tagged with it, in order to prevent unnecessary node rotations.
  • The SmoothGlue package allows currently-enabled SmoothGlue components to be disabled explicitly by setting the relevant _ENABLED Zarf flag to false.
  • The Zarf variables for enabling and disabling Istio-related components now work as expected again.
  • The Custom Resource Definitions for the latest version of External Secrets Operator are now applied using kubectl apply rather using Zarf's manifest component in order to avoid Helm issues with importing resources on existing clusters.
  • The SmoothGlue package no longer populates an empty gatewayCert if no Istio TLS certificate or key is provided.
  • The SmoothGlue package now skips the Keycloak CA cert gracefully if no Istio TLS cert is provided.
  • Overrides for the new Istio components will now be properly stored in component-specific Config Maps/Secrets, rather than as part of the the bigbang-overrides Config Map/Secrets.

6.16.0 (2025-05-29)​

🚨 Upgrade Notices​

  • Promtail has been deprecated, disabled by default, and replaced by Alloy.

    • If the user has any custom configurations of Promtail, they can utilize the Grafana Alloy migration utility to assist with the migration from Promtail to Alloy.
      • Follow the appropriate installation instructions to install Grafana Alloy.
      • With an active connection to the cluster containing a Promtail configuration, run:
        • kubectl get secret -n monitoring promtail-promtail -o yaml, or otherwise obtain the configuration from promtail.yaml.
        • Take the decoded contents of the promtail.yaml key and save locally: alloy convert --source-format=promtail --output=alloy.yaml promtail.yaml
        • Update any custom values.yaml references to .alloy.

  • The new variable, create_cloudwatch_log_group , defaults to true. To opt-out of importing and modifying the log group default retention time, set create_cloudwatch_log_group = false

    • ❗ For existing build clusters, it will require the import of the AWS CloudWatch Log Group (implicitly created by AWS) for each of the (typically 8) RDS instances (shown below) before running the Terraform IaC. Otherwise the IaC upgrade will fail.

    • If opting-in, then enter the following import commands for existing build clusters prior to configuring log retention and running the terraform IaC:

      terragrunt --terragrunt-working-dir modules/jira import 'module.rds.aws_cloudwatch_log_group.this["postgresql"]' /aws/rds/cluster/build-'dbIdentifier'-jira/postgresql
      terragrunt --terragrunt-working-dir modules/confluence import 'module.rds.aws_cloudwatch_log_group.this["postgresql"]' /aws/rds/cluster/build-'dbIdentifier'-confluence/postgresql
      terragrunt --terragrunt-working-dir modules/console import 'module.rds.aws_cloudwatch_log_group.this["postgresql"]' /aws/rds/cluster/build-'dbIdentifier'-console/postgresql
      terragrunt --terragrunt-working-dir modules/mattermost import 'module.rds.aws_cloudwatch_log_group.this["postgresql"]' /aws/rds/cluster/build-'dbIdentifier'-mattermost/postgresql
      terragrunt --terragrunt-working-dir modules/keycloak import 'module.rds.aws_cloudwatch_log_group.this["postgresql"]' /aws/rds/cluster/build-'dbIdentifier'-keycloak/postgresql
      terragrunt --terragrunt-working-dir modules/sonarqube import 'module.rds.aws_cloudwatch_log_group.this["postgresql"]' /aws/rds/cluster/build-'dbIdentifier'-sonarqube/postgresql

      # NOTE: The commands below are a bit different than those above.
      terragrunt --terragrunt-working-dir modules/gitlab import 'module.rds[0].module.db_instance.aws_cloudwatch_log_group.this["postgresql"]' /aws/rds/instance/build-'dbIdentifier'-gitlab/postgresql
      terragrunt --terragrunt-working-dir modules/nexus import 'module.rds[0].aws_cloudwatch_log_group.this["postgresql"]' /aws/rds/cluster/build-'dbIdentifier'-nexus/postgresql

  • The Auto SSO feature is now disabled by default. The following config can be added to the zarf-config.yaml file to enable the feature:

    package:
      deploy:
        set:
          AUTO_SSO_ENABLED: "true"

📦 SmoothGlue Features​

  • Promtail has been deprecated, disabled by default, and replaced by Alloy.
  • Kubernetes v1.31.x is officially supported and the default version used to test SmoothGlue on EKS/RKE2. Additional testing is performed for Kubernetes v1.32.x using internal single node instances on K3s.
  • SmoothGlue also now supports new native Istio Helm charts in preparation for the required migration off of Istio Operator. If users would like to test out the automation and new charts before they are required, as well as read about the pre-migration steps and migration concerns, please see the Istio Migration documentation.
  • IaC Version tracking has been added. Objects deployed via the IaC are saved as a variable in the commit_ref_name in the outputs and will show up as a tag on AWS objects with the tag sg:automation:commit-ref-name .
  • A new Terragrunt variable has been added in the HCL to allow setting apply_immediately for RDS modules. Setting this to true will apply Terraform changes to RDS instances to occur during IaC apply instead at scheduled maintenance time; the default remains false .
  • Auto SSO Feature:
    • This feature is now disabled by default. Please see the linked documentation below on how to enable and configure the feature.
    • SmoothGlue Run environments are now supported and new documentation on how to enable/configure the feature is now available.
  • The CloudWatch Log Group retention policy capability has been added for RDS/Aurora.
    • Two new variables, cloudwatch_log_group_retention_in_days and create_cloudwatch_log_group , have been added to application module inputs within the build.hcl file to configure the Aurora/RDS log group retention time, improving log retrieval times, overall system performance, and cost savings. (See upgrade notices for existing build clusters.)
    • The variable create_cloudwatch_log_group is defaulted to true. For existing clusters, see Upgrade Notices. For new clusters, use cloudwatch_log_group_retention_in_days to set retention days per database as seen below:
locals {
  gitlab_inputs = {
    # Adjust as needed, default is 0 days (logs never expire). Valid value for X is one of:
    # [0 1 3 5 7 14 30 60 90 120 150 180 365 400 545 731 1096 1827 2192 2557 2922 3288 3653]
    cloudwatch_log_group_retention_in_days = X

    # Set to false to avoid importing the cloudwatch_log_group for existing build clusters
    # create_cloudwatch_log_group = false
  }
  # Repeat for each module's input
  jira_inputs = {
    cloudwatch_log_group_retention_in_days = X
  }
  confluence_inputs = {...}
  keycloak_inputs = {...}
  mattermost_inputs = {...}
  sonarqube_inputs = {...}
  console_inputs = {...}
  nexus_inputs = {{...}
}
  • Grafana can be enabled with High Availability (HA). This creates multiple Grafana pods managed by an HPA with pod distribution rules and a backing RDS instance. See our docs for more information.
    • To enable Grafana HA, you must turn on the Grafana module, which can be completed by:
      • Setting the locals.grafana_inputs.high_availability to true
      • Setting modules.grafana to true
    • NOTE: Due to how Grafana pods need to communicate with each other to deconflict and de-duplicate, all database IaC features and in-cluster variables must be enabled at once at the Terragrunt level.
    • Grafana HA can be enabled on existing clusters, and maintainers should be aware of the following chart additions when enabling:
---
grafana:
  values:
    headlessService: true
    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 5
      targetCPU: "60"
      targetMemory: ""
    podDisruptionBudget:
      apiVersion: "policy/v1"
      minAvailable: 1
    grafana.ini:
      database:
        type: postgres
        host: "${db_host}:${db_port}"
        name: grafana
        user: "${db_name}"
      alerting:
        enabled: false
      unified_alerting:
        enabled: true
        ha_peers: monitoring-grafana-headless:9094
        ha_listen_address: ${POD_IP}:9094
        ha_advertise_address: ${POD_IP}:9094
        rule_version_record_limit: "5"
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: "kubernetes.io/hostname"
            labelSelector:
              matchLabels:
                dont-schedule-with: grafana

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.16.0 includes Big Bang Version 2.53.1. For more details on the features and updates included in Big Bang Version 2.53.1, please refer to the Big Bang Release Notes.
  • Confluence: chart bump(@2.0.0-bb.0) version bump confluence-node-lts:9.2.4
  • Jira: chart bump (2.0.0-bb.2)

🐞 Bug Fixes​

  • SmoothGlue now disables the KubeControllerManagerDown and KubeSchedulerDown Prometheus alerts by default for EKS since those components are located on the control-plane, which is managed by AWS and is not accessible from the cluster.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.54.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.31.8+rke2r1
    • K3s: v1.32.4+k3s1
    • EKS: v1.31.7
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.31.8-rke2r1-rocky-8-base-v1.1.1-stig-2025-05-19T08-22-30Z
    • EKS AMI: smoothglue-eks-1.31.7-rocky-8-base-v1.1.1-stig-2025-05-19T08-22-32Z
    • Base AMI: base-Rocky-8-EC2-LVM-v1.1.1-stig-2025-05-19T0702

6.15.0 (2025-05-14)​

🚨 Upgrade Notices​

  • Zarf is being updated to v0.54.0, which is now the minimum supported version. Trying to use an older Zarf version will result in Zarf registry failures on EKS clusters if using IRSA and S3 bucket backing for the registry (which are the default settings).
  • Ensure that you are using the new Zarf init config from the IaC. You will also need to run the Zarf init steps again to update Zarf. See the "Initializing Zarf on SmoothGlue" section in the "SmoothGlue Build/Run Deploy Guide" for more details. The Zarf registry will be unusable between the period from when the IaC is run and when Zarf is re-initialized.
ZARF_CONFIG=infra-iac/outputs/zarf-init-config.yaml zarf init --components git-server --architecture=amd64
  • The Nexus Repository Manager APIs changed impacting the blobstorage and repo jobs. We have temporarily modified the IaC values to disable these jobs and ensure the IaC values are applied, or the Nexus Repository Manager upgrade will fail.
  • You may need to restart the Keycloak pods using kubectl rollout restart statefulset -n keycloak keycloak to ensure that all pods are using the updated Keycloak theme bundled with this SmoothGlue release.

📦 SmoothGlue Features​

  • This release updates the styling and branding of the SmoothGlue Keycloak theme. Notably, the theme now includes a configurable Terms of Use banner, which can be configured on a per-realm basis. This feature is not enabled by default; to configure the Terms of Use banner, follow these steps:
    • Log into Keycloak's admin console and select the realm you wish to modify.
    • Navigate to "Realm Settings" -> "Localization" -> "Realm Overrides".
    • Click the "Add Translation" button to add a translation with the key "termsText". The value should be the text of your Terms of Use banner. This field supports HTML tags.
  • IAM Roles for Service Accounts (IRSA) is now enabled by default for EKS cluster nodes to access the Zarf registry, when backed up by an S3 bucket (by itself, a default setting). This moves away from the IMDSv1 based S3 bucket policy, which was a less secure access method. Existing clusters will transition seamlessly from IMDSv1 to IRSA based Zarf registry access without requiring any user intervention. Zarf has been updated to v0.54.0 to support this.

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.15.0 includes Big Bang Version 2.52.0. For more details on the features and updates included in Big Bang Version 2.52.0, please refer to the Big Bang Release Notes.
  • Upgrades the following Big Bang third-party apps:
    • Cert-Manager: 1.17.2
    • Confluence chart: 1.22.7-bb.1
    • Jira chart: 2.0.0-bb.0

🐞 Bug Fixes​

  • Updated the Keycloak theme, which resolved a Javascript error presented on the login page, as well as re-styled the "update password" page, which previously had white text on a white background for the password input field.

❗ Known Issues​

  • You may encounter a scenario following the upgrade or installation where istio-proxy fails to communicate properly with the istiod service. You may observe an error similar to the following:
    • To workaround this issue, restart the istiod Deployment.
2025-05-12T16:58:56.878375Z	warn	ca	ca request failed, starting attempt 4 in 804.379312ms
2025-05-12T16:58:57.683232Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing:

  • The following only applies to the initial deployment of the SmoothGlue IAC. No action is required for updates to already deployed clusters - Due to an upstream issue for the EKS module and when deploying a cluster using an AL2023 AMI, the System Integrator will need to manually generate and set a Zarf registry pull password. The following config can be added to the env.hcl file:

    locals {
      cluster_inputs = {
        zarf_registry_pull_password = "securepassword123"
      }
    }

  • When using a network load balancer (NLB) with the preserve_client_ip option enabled, the default routing rules for EKS nodes prevent nodes from accessing platform services hosted on the same node, which can cause failures when logging into Keycloak, particularly on clusters with fewer nodes.

    • More specifically, the default routing rules for nodes do not route traffic to the VPC router for traffic within the node’s local subnet, since these addresses should theoretically be reachable directly by the node. However, when using the preserve_client_ip option, the VPC router rewrites the source IP for traffic; when the node attempts to talk to the NLB, the traffic is rewritten so that it appears to come from the node itself, and the return traffic is not able to be routed correctly back to the NLB.
    • The following options are potential workarounds:
      • Disabling the preserve_client_ip option on the NLB will resolve the issue at the cost of losing source attribution for incoming traffic.
      • Removing the local subnet route on nodes will resolve the issue at the cost of increasing the amount and cost of traffic being routed through the VPC router.
      • Increasing the node count for the cluster will reduce the likelihood of the issue because it will become less likely for any given traffic to be routed back to the original node.

  • The Big Bang Istio Helm chart has a bug that prevents Istio Gateway deployments from properly being upgraded. During an upgrade, Istio Gateway deployments may get stuck as a result and will need manual intervention to complete the upgrade. To validate the issue in the cluster, check the health of the istiooperators.install.istio.io resource as follows:

    kubectl get istiooperators.install.istio.io -n istio-system

    If it is in Error status, delete all Istio Gateway deployments in the istio-system namespace to allow the Istio Operator to finish reconciling the upgrade and report a Healthy status. The deployments will be recreated automatically by the Istio Operator. For example:

    kubectl delete deployment.apps/admin-ingressgateway -n istio-system
    kubectl delete deployment.apps/passthrough-ingressgateway -n istio-system
    kubectl delete deployment.apps/public-ingressgateway -n istio-system

    Note: Deleting the deployments will entail some brief but non-zero downtime.

    https://repo1.dso.mil/big-bang/product/packages/istio-controlplane/-/issues/253 has been opened to track this issue, which was introduced in SmoothGlue 6.7 (Big Bang 2.44)

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.54.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.30.12+rke2r1
    • K3s: v1.32.3+k3s1
    • EKS: v1.30.9-eks-5d632ec

6.14.1 (2025-05-07)​

Package Bug Fixes​

  • add missing k8s-sidecar image (b2fc23e)

6.14.0 (2025-05-01)​

🚨 Upgrade Notices​

  • Updated Console's default config to change the default Keycloak host config from login.<domain> to keycloak.<domain>. SmoothGlue Build environments use the keycloak.<domain> host by default when deploying Keycloak. If deploying Keycloak under login.<domain> the following can be added to the bigbang-values.yaml to configure Console to use the overwritten host for Keycloak:

    packages:
      console:
        values:
          keycloak:
            host: login.<domain>

  • This upgrade includes a major version update to Keycloak. The full migration guide for Keycloak 26.0.0 is located here. Specific changes of note:

    • If you are currently setting the KC_PROXY environment variable to edge using the .addons.keycloak.values.secrets.env value, note that this option has been removed. It has been replaced by KC_PROXY_HEADERS option, which should be set automatically if the value .addons.keycloak.values.proxy.enabled is set to true.

      addons:
        keycloak:
          values:
            proxy:
              enabled: true

  • Keycloak may fail to upgrade.

    • To resolve this, reconcile the helm release and then delete the pods within the keycloak namespace.
      flux reconcile hr -n bigbang keycloak --with-source --force
      kubectl delete pods -n keycloak
    • Alternately, scale the keycloak pod replica count to 1 by editing the Horizontal Pod Autoscaler before starting the upgrade.
      kubectl patch hpa -n keycloak keycloak  -p "{\"spec\":{\"minReplicas\":1,\"maxReplicas\":1}}"

    And then scale it back up once done with the upgrade, for example.

      kubectl patch hpa -n keycloak keycloak  -p "{\"spec\":{\"minReplicas\":2,\"maxReplicas\":5}}"

  • The SmoothGlue EKS IaC's zarf-config output now includes Zarf variables for AWS and a variable for CLUSTER_NAME. When using the auto-SSO feature, please ensure the variables in the outputted zarf-config are merged with customer-managed zarf-config's to ensure Keycloak client names are configured properly.

📦 SmoothGlue Features​

  • SmoothGlue Automated SSO:
    • SmoothGlue now automatically configures the _structusureAdmins group with the appropriate Keycloak permissions to manage the smoothglue realm.
    • SmoothGlue now automatically configures cluster-level prefixes onto Keycloak clients managed by SmoothGlue. This change will enable future work to allow a SmoothGlue Run's SSO clients to be managed by SmoothGlue. Some applications may need to be manually restarted to pickup the new SSO client names.
    • SmoothGlue now automates the creation of the Keycloak objects for SonarQube. There are still some manual steps required by System Integrators to enable SSO for SonarQube. Please see updated documentation.
    • SmoothGlue will automatically configure a Keycloak client and automatically configure the application for:
      • Gitlab
      • Mattermost
      • Console
    • SmoothGlue will automatically configure a Keycloak client for:
      • Confluence
      • Jira
  • This release updates the styling and branding of the SmoothGlue Keycloak theme. Notably, the theme now includes a configurable terms of use banner which can be configured on a per-realm basis. This feature is not enabled by default; to configure the terms of use banner, follow these steps:
    • Log into Keycloak's admin console and select the realm you wish to modify.
    • Navigate to "Realm Settings" -> "Localization" -> "Realm Overrides".
    • Click the "Add Translation" button to add a translation with the key "termsText". The value should be the text of your Terms of Use banner. This field supports HTML tags.

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.14.0 includes Big Bang Version 2.51.0. For more details on the features and updates included in Big Bang Version 2.51.0, please refer to the Big Bang release notes.
    • Kiali stays pinned to the earlier 2.50.0 version of v2.6.0 due to a failure seen in latest version.

🐞 Bug Fixes​

  • Docusaurus and Tailwind config to support dark mode by default. Updated global footer to add a label for the legal links.

❗ Known Issues​

  • The following only applies to the initial deployment of the SmoothGlue IAC. No action is required for updates to already deployed clusters - Due to an upstream issue for the EKS module and when deploying a cluster using an AL2023 AMI, the System Integrator will need to manually generate and set a Zarf registry pull password. The following config can be added to the env.hcl file:

    locals {
      cluster_inputs = {
        zarf_registry_pull_password = "securepassword123"
      }
    }

  • When using a network load balancer (NLB) with the preserve_client_ip option enabled, the default routing rules for EKS nodes prevent nodes from accessing platform services hosted on the same node, which can cause failures when logging into Keycloak, particularly on clusters with fewer nodes.

    • More specifically, the default routing rules for nodes do not route traffic to the VPC router for traffic within the node’s local subnet, since these addresses should theoretically be reachable directly by the node. However, when using the preserve_client_ip option, the VPC router rewrites the source IP for traffic; when the node attempts to talk to the NLB, the traffic is rewritten so that it appears to come from the node itself, and the return traffic is not able to be routed correctly back to the NLB.
    • The following options are potential workarounds:
      • Disabling the preserve_client_ip option on the NLB will resolve the issue at the cost of losing source attribution for incoming traffic.
      • Removing the local subnet route on nodes will resolve the issue at the cost of increasing the amount and cost of traffic being routed through the VPC router.
      • Increasing the node count for the cluster will reduce the likelihood of the issue because it will become less likely for any given traffic to be routed back to the original node.

  • The Big Bang Istio Helm chart has a bug that prevents Istio Gateway deployments from properly being upgraded. During an upgrade, Istio Gateway deployments may get stuck as a result and will need manual intervention to complete the upgrade. To validate the issue in the cluster, check the health of the istiooperators.install.istio.io resource as follows:

    kubectl get istiooperators.install.istio.io -n istio-system

    If it is in Error status, delete all Istio Gateway deployments in the istio-system namespace to allow the Istio Operator to finish reconciling the upgrade and report a Healthy status. The deployments will be recreated automatically by the Istio Operator. For example:

    kubectl delete deployment.apps/admin-ingressgateway -n istio-system
    kubectl delete deployment.apps/passthrough-ingressgateway -n istio-system
    kubectl delete deployment.apps/public-ingressgateway -n istio-system

    Note: Deleting the deployments will entail some brief but non-zero downtime.

    https://repo1.dso.mil/big-bang/product/packages/istio-controlplane/-/issues/253 has been opened to track this issue, which was introduced in SmoothGlue 6.7 (Big Bang 2.44)

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.46.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.30.11+rke2r1
    • K3s: v1.32.3+k3s1
    • EKS: v1.30.9-eks-5d632ec
  • Refer to the SmoothGlue documentation for additional guidance.
  • For details on the Big Bang release, see the Big Bang Release Notes.

6.13.0 (2025-04-16)​

📦 SmoothGlue Features​

  • A new optional variable cloudwatch_log_group_retention_in_days has been added to the env.hcl files to configure the EKS cluster log group retention time. It can be configured as shown below.

    locals {
      cluster_inputs = {
        # Adjust as needed, default is 90 days. Valid value for X is one of:
        # [0 1 3 5 7 14 30 60 90 120 150 180 365 400 545 731 1096 1827 2192 2557 2922 3288 3653]
        cloudwatch_log_group_retention_in_days = X
      }
    }

  • IAM policies generated by the SmoothGlue IAC no longer apply tags to IAM policies when compatibility_mode is set to true. This change is to conform to limitations on high-side deployments

    • NOTE: On an existing cluster, you may need to delete the allow_kms and allow_cluster_autoscaler IAM policies to allow their recreation.

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.13.0 includes Big Bang Version 2.50.0. For more details on the features and updates included in Big Bang Version 2.50.0, please refer to the Big Bang release notes.
  • Upgrades the following Big Bang third-party apps:
    • Confluence LTS: 9.2.3
    • Jira LTS: 10.3.5
    • Nexus IQ Server: 1.189.0-01

🐞 Bug Fixes​

  • Fixed an issue that prevented user data scripts from running on AL2023 AMIs.
  • Fixed an issue that overwrote default values from SmoothGlue with customer overrides. This prevented SmoothGlue default values from being viewable at runtime. Overrides provided by customers are still overlayed on top of SmoothGlue default values, so this is a purely cosmetic change.

❗ Known Issues​

  • The following only applies to the initial deployment of the SmoothGlue IAC. No action is required for updates to already deployed clusters - Due to an upstream issue for the EKS module and when deploying a cluster using an AL2023 AMI, the System Integrator will need to manually generate and set a Zarf registry pull password. The following config can be added to the env.hcl file:

    locals {
      cluster_inputs = {
        zarf_registry_pull_password = "securepassword123"
      }
    }

  • When using a network load balancer (NLB) with the preserve_client_ip option enabled, the default routing rules for EKS nodes prevent nodes from accessing platform services hosted on the same node, which can cause failures when logging into Keycloak, particularly on clusters with fewer nodes.

    • More specifically, the default routing rules for nodes do not route traffic to the VPC router for traffic within the node’s local subnet, since these addresses should theoretically be reachable directly by the node. However, when using the preserve_client_ip option, the VPC router rewrites the source IP for traffic; when the node attempts to talk to the NLB, the traffic is rewritten so that it appears to come from the node itself, and the return traffic is not able to be routed correctly back to the NLB.
    • The following options are potential workarounds:
      • Disabling the preserve_client_ip option on the NLB will resolve the issue at the cost of losing source attribution for incoming traffic.
      • Removing the local subnet route on nodes will resolve the issue at the cost of increasing the amount and cost of traffic being routed through the VPC router.
      • Increasing the node count for the cluster will reduce the likelihood of the issue because it will become less likely for any given traffic to be routed back to the original node.

  • The Big Bang Istio Helm chart has a bug that prevents Istio Gateway deployments from properly being upgraded. During an upgrade, Istio Gateway deployments may get stuck as a result and will need manual intervention to complete the upgrade. To validate the issue in the cluster, check the health of the istiooperators.install.istio.io resource as follows:

    kubectl get istiooperators.install.istio.io -n istio-system

    If it is in Error status, delete all Istio Gateway deployments in the istio-system namespace to allow the Istio Operator to finish reconciling the upgrade and report a Healthy status. The deployments will be recreated automatically by the Istio Operator. For example:

    kubectl delete deployment.apps/admin-ingressgateway -n istio-system
    kubectl delete deployment.apps/passthrough-ingressgateway -n istio-system
    kubectl delete deployment.apps/public-ingressgateway -n istio-system

    Note: Deleting the deployments will entail some brief but non-zero downtime.

    https://repo1.dso.mil/big-bang/product/packages/istio-controlplane/-/issues/253 has been opened to track this issue, which was introduced in SmoothGlue 6.7 (Big Bang 2.44)

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.46.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.8+rke2r1
    • K3s: v1.32.3+k3s1
    • EKS: v1.30.9-eks-5d632ec

6.12.0 (2025-04-02)​

🚨 Upgrade Notices​

  • Kyverno
    • A new Kyverno Policy has been added which mutates pod specs to drop ALL capabilities in all containers if not already done. This policy works in tandem with the require-drop-all-capabilities policy to make it easier for SREs to securely deploy workloads to their clusters without having to explicitly modify the pod's containers' securityContexts to be compliant.
    • If Big Bang consumers are currently excluding certain workloads from the require-drop-all-capabilities policy due to incompatibilities with that policy, those exclusions should also be included for this new policy: add-default-capability-drop to avoid workload interruption.

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.12.0 includes Big Bang Version 2.49.0. For more details on the features and updates included in Big Bang Version 2.49.0, please refer to the Big Bang Release Notes.
  • console updated image to 54183
  • nexus-iq chart upgraded to 188

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.46.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.30.9-rke2r1
    • K3s: v1.30.9+k3s1
    • EKS: v1.30.8
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.30.9-rke2r1-rocky-8-base-v1.1.1-stig-2025-02-17T09-24-30Z
    • EKS AMI: smoothglue-eks-1.30.8-rocky-8-base-v1.1.1-stig-2025-02-10T09-21-19Z
    • Base AMI: base-Rocky-8-EC2-LVM-v1.1.1-stig-2025-02-10T0802

6.11.0 (2025-03-19)​

🚨 Upgrade Notices​

  • Flux has been upgraded to 2.5.1. Platform Operators should update their local Flux binary to a compatible version.

📦 SmoothGlue Features​

  • On RKE2-based clusters, the preserve_client_ips IaC variable is now set to false by default in order to allow pods to communicate internally using external DNS names. See the known issues section for more information.
    Note: This means that the client IP in any logs will appear to be the load balancer itself. To get the true client IP, you will need to setup monitoring on the NLB
  • Added documentation for manually rotating RDS/Aurora database passwords as a good security practice.

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.11.0 includes Big Bang Version 2.48.0. For more details on the features and updates included in Big Bang Version 2.48.0, please refer to the Big Bang Release Notes.
  • Update Gitlab to 17.9.2 (applied Critical Patch)
  • Update Jira to LTS 10.3.4 (addresses CVE-2024-38819)
  • Update JSM to 10.3.4 (addresses CVE-2024-38819)

🐞 Bug Fixes​

  • Fixed an issue with the SmoothGlue automated SSO feature for ArgoCD. SmoothGlue Admins should now be correctly given admin privileges in ArgoCD.

❗ Known Issues​

  • When using a network load balancer (NLB) with the preserve_client_ip option enabled, the default routing rules for EKS nodes prevent nodes from accessing platform services hosted on the same node, which can cause failures when logging into Keycloak, particularly on clusters with fewer nodes.
    • More specifically, the default routing rules for nodes do not route traffic to the VPC router for traffic within the node’s local subnet, since these addresses should theoretically be reachable directly by the node. However, when using the preserve_client_ip option, the VPC router rewrites the source IP for traffic; when the node attempts to talk to the NLB, the traffic is rewritten so that it appears to come from the node itself, and the return traffic is not able to be routed correctly back to the NLB.
    • The following options are potential workarounds:
      • Disabling the preserve_client_ip option on the NLB will resolve the issue at the cost of losing source attribution for incoming traffic.
      • Removing the local subnet route on nodes will resolve the issue at the cost of increasing the amount and cost of traffic being routed through the VPC router.
      • Increasing the node count for the cluster will reduce the likelihood of the issue because it will become less likely for any given traffic to be routed back to the original node.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.46.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.30.9-rke2r1
    • K3s: v1.30.9+k3s1
    • EKS: v1.30.8
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.30.9-rke2r1-rocky-8-base-v1.1.1-stig-2025-02-17T09-24-30Z
    • EKS AMI: smoothglue-eks-1.30.8-rocky-8-base-v1.1.1-stig-2025-02-10T09-21-19Z
    • Base AMI: base-Rocky-8-EC2-LVM-v1.1.1-stig-2025-02-10T0802

6.10.0 (2025-03-04)​

SmoothGlue Features​

  • This release adds optional basic support for Amazon Linux 2023 (AL2023) AMIs in EKS cluster IaC. To use AL2023, add the following to the cluster_inputs section of your env.hcl file: 
    locals {
      cluster_inputs = {
        ami_id           = "ami-0123456789abcdef0"  # replace with actual AMI ID
        default_ami_type = "AL2023_x86_64_STANDARD"
      }
    }
  • This release adds optional support in IaC for provisioning GitLab's database using an RDS Multi-AZ cluster rather than a single database instance. As a single instance, RDS offers support for a single warm standby instance; however, provisioning the database using an RDS Multi-AZ cluster allows for a cluster of three instances. There is no automatic migration path from a single RDS instance to a Multi-AZ cluster, so we recommend enabling the Multi-AZ cluster during the initial cluster provision if possible. If migrating an existing cluster, you will need to perform a database import/export manually.
    • Note that there are instance class limitations when using a Multi-AZ RDS cluster; see the AWS documentation for more information.

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.10.0 includes Big Bang Version 2.47.0. For more details on the features and updates included in Big Bang Version 2.47.0, please refer to the Big Bang Release Notes.
  • Update Confluence to LTS 9.2.1 (Helm chart version 1.22.5-bb.0).
  • Update Jira to LTS 10.3.3 (Helm chart version 1.22.5-bb.1).

🪲 Bug Fixes​

  • Refactor IaC compatibility mode toggle to correctly disable NLB stickiness in ISO regions.
  • Exclude aws-ebs-csi-driver namespace from generate-networkpolicy-imds Kyverno ClusterPolicy so that RKE2 clusters can provision EBS PVCs correctly.
  • Exclude the following namespaces from the require-istio-on-namespaces Kyverno ClusterPolicy so that users may enforce the policy:
    • cluster-autoscaler
    • crossplane-system
    • kyverno
    • structsure-system
  • Extend HelmRelease install timeout for GitLab to 15 minutes.

❗️ Known Issues​

  • When using a network load balancer (NLB) with the preserve_client_ip option enabled, the default routing rules for EKS and RKE2 nodes prevent nodes from accessing platform services hosted on the same node, which can cause failures when logging into Keycloak, particularly on clusters with fewer nodes.
    • More specifically, the default routing rules for nodes do not route traffic to the VPC router for traffic within the node's local subnet, since these addresses should theoretically be reachable directly by the node. However, when using the preserve_client_ip option, the VPC router rewrites the source IP for traffic; when the node attempts to talk to the NLB, the traffic is rewritten so that it appears to come from the node itself, and the return traffic is not able to be routed correctly back to the NLB.
    • We are currently working on an Istio-level fix which should prevent VirtualService traffic within the cluster from ever leaving the cluster. Until that fix is available, the following options are potential workarounds:
      • Disabling the preserve_client_ip option on the NLB will resolve the issue at the cost of losing source attribution for incoming traffic.
      • Removing the local subnet route on nodes will resolve the issue at the cost of increasing the amount and cost of traffic being routed through the VPC router.
      • Increasing the node count for the cluster will reduce the likelihood of the issue because it will become less likely for any given traffic to be routed back to the original node.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.46.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.30.9-rke2r1
    • K3s: v1.30.9+k3s1
    • EKS: v1.30.8
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.30.9-rke2r1-rocky-8-base-v1.1.1-stig-2025-02-17T09-24-30Z
    • EKS AMI: smoothglue-eks-1.30.8-rocky-8-base-v1.1.1-stig-2025-02-10T09-21-19Z
    • Base AMI: base-Rocky-8-EC2-LVM-v1.1.1-stig-2025-02-10T0802

🔗 Helpful Links​

6.9.0 (2025-02-19)​

🚨 Upgrade Notices​

  • During upgrade, you may get a SonarQube is under maintenance error message on the SonarQube UI.
    • To resolve this, once the HelmRelease upgrades, you will be prompted to visit your SonarQube instance at a <sonarqube_url>/setup URL.

📦 SmoothGlue Features​

  • Crossplane Upgraded the Crossplane and provider-kubernetes Crossplane components.
  • IaC: Added HA support for RDS Aurora modules:
    • Supported Applications:
      • Jira
      • Confluence
      • Mattermost
      • SonarQube
      • Nexus
      • Console
      • Keycloak
    • For any of the above modules, you can now add more than one RDS instance into a cluster. Additional instances will be Reader instances only. If the main Writer instance goes down, Aurora will automatically promote a Reader instance to Writer.
      • For each instance created, values such as the availability zone can be manually set; however, you do not have to specify AZ for each instance; Aurora will automatically place each instance in a different AZ.
      • All RDS Aurora storage is automatically replicated across multiple AZs regardless of DB instance count.
      • Examples

        • To create a writer instance and two reader instances for keycloak in your env.hcl:

          keycloak_inputs = {
            # Allows a specific number of database instances to be defined
            rds_instances = {
              primary     = {availibility_zone = us-east-1a}
              secondary   = {}
              replica1    = {}
              # ...
            } 
          }

        • Autoscaling of instances is also optionally available. Aurora autoscaling will NOT scale any instances explicitly defined in rds_instances; it will only add or remove reader instances up to the defined min and max limits. Autoscaling will use the target_metric scaling policy by default with a target CPU utilization of 70%. The following env.hcl provisions Keycloak RDS Aurora autoscaling with between 0 and 5 reader instances:

          keycloak_inputs = {
            rds_auto_scale = {
              enabled = true
              min = 0 # default is 0
              max = 5 # default is 5
            }
          }

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.9.0 includes Big Bang Version 2.46.0. For more details on the features and updates included in Big Bang Version 2.46.0, please refer to the Big Bang release notes.
  • Confluence: confluence-node:9.2.0 version: 1.22.3-bb.4
    • Removed duplicate jmx-initContainer
    • Updated cypress (source) 14.0.0 -> 14.0.1
  • Jira: jira-node-lts:10.3.2. version: 1.22.3-bb.0
    • Updated chart to 1.22.3
    • Updated cypress (source) 14.0.0 -> 14.0.1
  • Nexus IQ: Upgraded from 1.186.0-01 to 1.187.0-01
  • Crossplane Components:
    • crossplane - v1.16.0 to v1.19.0
    • provider-kubernetes - v0.12.1 to 0.16.2

❗ Known Issues​

  • If turning on new components, Zarf health checks are performed before unsuspending Big Bang. Manually resume the Big Bang HelmRelease, as required.
  • Big Bang 2.46.0 comes with a known issue relating to the gitlab-gitlab-exporter ServiceMonitor object. We are handling this issue as part of our upgrade process; no user action should be required. More information may be found here.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.46.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.30.9+rke2r1
    • K3s: v1.31.5+k3s1
    • EKS: v1.30.8-eks-2d5f260
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.30.9-rke2r1-rocky-8-base-v1.1.1-stig-2025-02-17T09-24-30Z
    • EKS AMI: smoothglue-eks-1.30.6-rocky-8-base-v1.1.1-stig-2025-01-04T03-12-26Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

6.8.0 (2025-02-06)​

🚨 Upgrade Notices​

  • SmoothGlue packages are now built with Zarf v0.46.0, which is the minimum version supported. Please zarf init pre-existing clusters with the v0.46.0 init package before upgrading SmoothGlue.

  • The new Zarf version provides better package readiness checking. As a byproduct, the logic in the package has less control over when and what is evaluated. The default readiness timeout set by Zarf is too low for deploying a fresh cluster. It is recommended to add the following to the ZARF_CONFIG file:

    package:
      deploy:
        timeout: 30m0s

  • Due to the better readiness checks from Zarf, clusters that do not wish to use the automated SSO feature need to disable it from the config. Run clusters have it disabled by default, but for build clusters it is recommended to include the following to the ZARF_CONFIG file to opt out of the automated SSO feature:

    package:
      deploy:
        set:
          KEYCLOAK_CONFIG_ENABLED: false

  • This release will cause a node refresh to occur.

📦 SmoothGlue Features​

  • IaC Allow overriding EKS-calculated max-pods per node.

⏩ Upgraded Packages​

  • Upgraded Zarf to v0.46.0
  • Upgraded Confluence to confluence-node:9.2.0 version: 1.22.3-bb.2
    • Updated gluon from 0.5.12 to 0.5.14
    • Updated cypress dependencies 13.12.0 -> ^14.0.0
    • Updated registry1.dso.mil/ironbank/opensource/postgres/postgresql from 16.6 to 17.2
  • Upgraded Jira to jira-node-lts:10.3.2 version: 1.22.2-bb.4
    • Added gluon 0.5.12 -> 0.5.14
    • Updated cypress ^13.15.0 -> ^14.0.0
    • Updated registry1.dso.mil/ironbank/atlassian/jira-data-center/jira-node-lts 10.3.1 -> 10.3.2
  • This release of SmoothGlue Enterprise v6.8.x includes Big Bang Version 2.45.1. For more details on the features and updates included in Big Bang Version 2.45.1, please refer to the Big Bang Release Notes.
    • Promtail: Note: bumping promtail image/appVersion beyond the version used in upstream chart (v3.0.0 vs v3.3.2)
    • Mattermost upgrade from 10.4.1 to 10.4.2
    • GitLab upgrade from 17.6.2 to 17.8.1

🪲 Bug Fixes​

  • Standardize Terraform provider versions to resolve lookup inconsistencies.
  • Nexus can be enabled with nexus = true or nexusRepositoryManager = true, allowing for conditional enablement of Nexus Repository Manager.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.46.0.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.30.8-rke2r1
    • K3s: v1.30.5+k3s1
    • EKS: v1.30.8-eks-2d5f260
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.30.8-rke2r1-rocky-8-base-v1.1.1-stig-2025-01-13T09-22-54Z
    • EKS AMI: smoothglue-eks-1.30.6-rocky-8-base-v1.1.1-stig-2025-01-04T03-12-26Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

🔗 Helpful Links​

6.7.0 (2025-01-22)​

📦 SmoothGlue Features

  • Kubernetes v1.30.x is officially supported and is the default version used to test SmoothGlue on EKS/RKE2. Additional testing is performed for Kubernetes v1.31.x using K3s.
  • IaC: allow autoscaling on a per-nodegroup basis with supporting documentation. Cluster autoscaler will be enabled by default on the main nodegroup. Additional nodegroups can be explicitly defined via tags.

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.7.0 includes Big Bang Version 2.44.0. For more details on the features and updates included in Big Bang Version 2.44.0, please refer to the Big Bang release notes.
  • console updated image to 39560
  • nexus-iq chart upgraded to 186
  • cluster-autoscaler upgrade to support Kubernetes v1.30.x

❗ Known Issues​

  • Kiali - ISSUE

    • On Kubernetes 1.29+, the Kiali Operator may fail with a 404 while running the kiali-deploy playbook if the cluster returns the flowcontrol.apiserver.k8s.io/v1beta2 API version (no longer served as of v1.29).

      In this case, removing the invalid API version should resolve the issue and allow the Kiali Operator to run successfully.

 $ kubectl delete apiservices.apiregistration.k8s.io v1beta2.flowcontrol.apiserver.k8s.io

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.36.1.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.30.8-rke2r1
    • K3s: v1.30.5+k3s1
    • EKS: v1.30.8-eks-2d5f260
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.30.8-rke2r1-rocky-8-base-v1.1.1-stig-2025-01-13T09-22-54Z
    • EKS AMI: smoothglue-eks-1.30.6-rocky-8-base-v1.1.1-stig-2025-01-04T03-12-26Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

6.6.0 (2025-01-07)​

🚨 Upgrade Notices​

  • :octagonal_sign: With a Major version update to Jira 10.3 you must also update the SSO addon, this is not provided for you if you are running Jira in a disconnected environment.

📦 SmoothGlue Features​

  • Adds Grafana Dashboard / Alerts for monitoring failed Keycloak login attempts by Username and IP
  • Jira has a major version update that changes how users SSO login, To force users to have to login again see this guide

⏩ Upgraded Packages​

  • This release of SmoothGlue Enterprise v6.6.0 includes Big Bang Version 2.43.0. For more details on the features and updates included in Big Bang Version 2.43.0, please refer to the Big Bang release notes.
    • Jira has received a major version to 10

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.36.1.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.8+rke2r1
    • K3s: v1.30.5+k3s1
    • EKS: v1.29.6
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.29.8-rke2r1-rocky-8-base-v1.1.1-stig-2024-09-23T08-14-20Z
    • EKS AMI: smoothglue-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-09-09T08-14-46Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

6.5.0 (2024-12-30)​

🚨 Upgrade Notices​

If you see a :octagonal_sign: it means that some form of manual step is required to proceed, please heed these warnings.

  • :octagonal_sign: Zarf version required is now v0.36.1 to support new functionality around deploying OCI artifacts. Upgrading existing clusters requires using the new version to zarf init the cluster to upgrade onto the new version
  • :octagonal_sign: Due to a FIPs compliance issue in the Big Bang's version of Gitlab you MUST upgrade the RDS for GitLab from Postgres version 14 to version 16 while staying on the same version of GitLab. It is recommended to upgrade to Postgres 16 before attempting to upgrade via the IaC. Steps to manually upgrade GitLab RDS:
    • Fully backup GitLab and store backup in secure location
    • Scale down GitLab deployments and statefulsets
    • Go to AWS console and find your GitLab instance (ity won't be in a cluster)
    • Click Modify in the top right
    • Change DB engine version to 16.X
    • Scroll to "additional configurations" --> "Database options"
    • Change DB parameter group to default.postgres16
    • Click continue and BE SURE TO SELECT Apply Immediately
      • AWS will take ~10 minutes to upgrade. Please make sure the RDS is done upgrading before proceeding.
    • Now run the IaC for 6.5. The IaC should accept the database engine version 16 and create a new aws_db_parameters with the 16 family
  • :octagonal_sign: The Terraform EKS module and a major change to how roles attach to the Cluster has been implemented.
    • If you encounter an error in the Terragunt/Terraform with the object called aws_eks_access_entry due to the object already existing, you must:
    • terragrunt import 'module.eks.aws_eks_access_entry.this["<Your access entry name In the HCL>"]' <cluster name>:arn:aws:iam::<aws account>:role/ec2/<Role id>
  • :octagonal_sign: This release updates the Terraform module for AWS EKS from major version 19 to 20, which enables support for EKS cluster access entries. We recommend migrating from the aws-auth ConfigMap to cluster access entries, as this will become the preferred authentication mode for EKS clusters moving forward.

    • For a role which was previously defined using the following parameter in the env.hcl:

      aws_auth_roles = [
        {
          rolearn  = "arn:aws:iam::012345678901:role/AWSReservedSSO_AdministratorAccess_0123456789abcdef",
          username = "AWSReservedSSO_AdministratorAccess_0123456789abcdef",
          groups = [
            "system:masters",
          ]
        },
      ]

      The following access entry is equivalent:

      access_entries = {
        admin = {
          principal_arn = "arn:aws:iam::012345678901:role/aws-reserved/sso.amazonaws.com/us-east-2/AWSReservedSSO_AdministratorAccess_0123456789abcdef"
          policy_associations = {
            cluster_admin = {
              policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
              access_scope = {
                type = "cluster"
              }
            }
          }
        }
      }

    • For existing clusters which are migrating to the API_AND_CONFIG_MAP authentication method, an existing access entry for the cluster creator will be exposed during the migration. was previously not visible when using aws-auth ConfigMap, but will become visible when access entry is enabled. If you are defining a cluster access entry for this IAM entity, it must be imported into Terraform using the following command:

      terragrunt import 'module.eks.aws_eks_access_entry.this["<access_entry_key"]' <eks_cluster_name>:<arn_of_iam_entity>

    • By default, the created EKS clusters will enable authentication via both the newly-enabled cluster access entries, as well as the legacy aws-auth ConfigMap.

      • If you are relying upon the aws-auth ConfigMap in an existing cluster, note that due to major version 20 of the EKS module removing the aws-auth functionality from the core of the module, the aws-auth ConfigMap is created using a separate submodule. This means the existing ConfigMap will be re-created during the Terraform apply process, and any users whose permissions are defined only in the aws-auth ConfigMap may temporarily lose access to the cluster until the ConfigMap is re-created. This should be a one-time process.

    • This version adds the following new variables to the eks-cluster Terraform module:

      • authentication_mode: Determines the enabled EKS authentication modes, defaults to (API_AND_CONFIG_MAP).
      • access_entries: A map of the cluster access entries for the cluster, see the example env.hcl for more information.
      • enable_cluster_creator_admin_permissions: Automatically creates a cluster access entry for the identity running the Terraform module. (This should not be set to true if an explicit access entry is being created for this identity).
  • :octagonal_sign: Due to a Big Bang update for Kyverno 1.13.0 that deprecated how cluster policies are generated and the fact that cluster policies are immutable; the old cluster policies must be manually deleted to allow for the same policies to be recreated. If you have your own custom cluster policies that used generate-XYZ please see Upstream Kyverno release notes to ensure that they follow the new standards. https://kyverno.io/blog/2024/10/30/announcing-kyverno-release-1.13/
    • kubectl delete clusterpolicy generate-networkpolicy-imds
    • kubectl delete clusterpolicy generate-private-git-server-secret
  • ❗ In kyverno update to v1.13 they have remove wildcard permissions which allowed Kyverno controllers to view all resources. We have added back in the wildcard permissions for the time being but all users should follow best practices and remove them
  • ❗ The EKS terraform module update will cause a node rotation, this can take a long time depending on your availability settings and terraform might time out until the EKS node group is back into an active state
  • ❗ GitLab HR will succeed but there will be a job that fails called gitlab-gitlab-upgrade-check, you can delete the job as it is just a warning

📦 SmoothGlue Features​

  • iac: enable specifying EKS cluster log types to save to cloudwatch
  • iac: update eks tf module, support eks cluster access entries

⏩ Upgraded Packages​

🐞 Bug Fixes​

  • This fix restores RKE2 functionality (NeuVector is not working), which has been broken since SmoothGlue release 6.2.0.
  • The AWS EFS CSI driver add-on has been locked down to version v2.1.0 for EKS. The latest version v2.1.1 gives EFS mount failure issues with NeuVector, Jira and Confluence.
  • zarf: add deny imds exclusion for aws-efs-csi-driver

❗ Known Issues​

  • There is a chance that the Kiali pod will be stuck in a non functional state, rotate the pod and it should fix itself
  • crossplane-provider-keycloak might be stuck in an unhealthy state, to remedy find the providerrevision.pkg.crossplane.io for the crossplane-provider-keycloak and force delete it so it can be recreated. This is will be for both run and build.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.36.1.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.8+rke2r1
    • K3s: v1.30.5+k3s1
    • EKS: v1.29.6
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.29.8-rke2r1-rocky-8-base-v1.1.1-stig-2024-09-23T08-14-20Z
    • EKS AMI: smoothglue-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-09-09T08-14-46Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

Other Changes​

  • console: deploy chart from OCI artifact (d3f1c74)**

6.4.0 (2024-12-13)​

🚨 Upgrade Notices​

  • For SmoothGlue users using EKS you must go into AWS Console and manually pin aws-efs-csi-driver
    1. Log into AWS console --> EKS
    2. Navigate to your cluster
    3. Got to the add-ons section
    4. Search for "Amazon EFS CSI Driver"
    5. Edit and select version v2.1.0-eksbuild.1

📦 SmoothGlue Features​

  • Add ALB support as an optional module that users can leverage instead of the default load balancer.
  • Allow setting S3 block_public access. A new IaC flag block_public_access has been added to the env.hcl files in the infra-iac/envs/ directory of the AWS IaC repository. If set to true (the default), this flag blocks all public access to S3 buckets created for the cluster.

⏩ Upgraded Packages​

  • Nexus IQ upgrade to 184
    • Refer to https://help.sonatype.com/en/iq-2024-release-notes.html#idp212975 for more details. If you enabled the Golden Versions feature in release 183 and then upgraded IQ server to 184, you will need to disable and re-enable the feature post-upgrade in order to access it. This is a one-time requirement and will not be required after your next upgrade.
  • This release of SmoothGlue Enterprise v6.4.0 includes Big Bang Version 2.41.0. For more details on the features and updates included in Big Bang Version 2.41.0, please refer to the Big Bang Release Notes.
    • Kiali - MR:
      • By default, Kiali has access to all namespaces within a given cluster. However, if you have restricted access to only specific namespaces, please review this note prior to upgrading, as the methodology behind this has changed.
    • External-secrets - MR:
      • If you are deploying any deprecated v1alpha1 custom resources, your deployment may break with this patch. Please upgrade resources to non-deprecated versions.
    • BBTOC- MR
      • In an effort to provide more clarity on where each package stands within Big Bang, we have implemented the Package Maintenance Tracks as approved by the BBTOC. This should provide more information on how different packages are maintained and tested. In order to facilitate this, each package that is maintained & integrated by Big Bang (not community maintained packages) now has a badge added on the readme to identify what track each package is on.

🐞 Bug Fixes​

  • Adjust asg_attachment module logic to support optional ALB.
  • Add runtimePath for NeuVector Enforcer.

❗️ Known Issues​

  • NeuVector Helm Release fails on RKE2 clusters due to an AWS EFS CSI driver issue.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.32.6.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.9+rke2r1
    • K3s: v1.31.3+k3s1
    • EKS: v1.29.8
  • The following AMI versions were used for testing:
    • RKE2 AMI: structsure-rke2-v1.29.9-rke2r1-rocky-8-base-v1.1.1-stig-2024-10-28T08-12-25Z
    • EKS AMI: structsure-eks-1.29.8-rocky-8-base-v1.1.1-stig-2024-10-28T08-12-34Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

6.3.0 (2024-11-25)​

🚨 Upgrade Notices​

  • PostgreSQL 13 is no longer a supported version for Confluence 9.1.x. For this Confluence version, you must upgrade to at least RDS 14.x. Applying the IaC for this version will upgrade Confluence's database to 14.x. As such, if you are running Confluence, ensure you run the IaC before upgrading the package on the cluster.
    • Prior to applying the IaC for the RDS upgrade, suspend the Confluence helm release and scale the Confluence statefulset to 0. The HR can be resumed after the 14.x RDS is available and healthy.
    • Upon visiting the login screen, users may be prompted with a database thread warning. Click Accept to continue.
    • SSO will be disabled on initial login due to a miniOrange upgrade dependency. Log in with admin credentials, and upgrade miniOrange to 2.3.2 in the Manage Apps section of the Admin panel.

📦 SmoothGlue Features​

  • Kyverno Policies

    • A new policy named generate-networkpolicy-imds has been added to the default Kyverno policies. This Kyverno policy will generate a network policy in any non-Big Bang namespace. The network policy will block egress traffic to IMDS. This policy can be disabled by adding the following to the Big Bang values:

      kyvernoPolicies:
        values:
          additionalPolicies:
            generate-networkpolicy-imds:
              enabled: false
  • Crossplane provider-gitlab

    • provider-gitlab has been enabled by default. It allows Crossplane to automate functions in GitLab. This provider is used by Console to enable project creation, initialization, and manage project settings to enforce a common baseline. Currently, provider-gitlab requires additional steps to enable automation within GitLab. provider-gitlab can be disabled by adding the following to the zarf-deploy-config file:

      package:
        deploy:
          components: '-crossplane-provider-gitlab'

⏩ Upgraded Packages​

  • Jira upgrade to 9.12.15
  • Confluence upgrade to 9.1.1
    • End of support for PostgreSQL 13; provides an upgrade to PostgreSQL 14.x
    • Java 21 bundled with Confluence
      • Eclipse Temurin Java 21 is now included with Confluence installations and upgrades via the installer
    • Dark theme support for custom logos and color schemes
    • Fixes multiple CVEs
    • Issues Resolved
    • Full Release Notes
  • This release of SmoothGlue Enterprise v6.3.0 includes Big Bang Version 2.40.0. For more details on the features and updates included in Big Bang Version 2.40.0, please refer to the Big Bang Release Notes.

    • Istio-controlplane:

      • This release adds a default EnvoyFilter to increase the security of the Istio cluster. This filter, which defaults to enabled, can be disabled using e.g., istio.Values.defaultSecurityHeaders.enabled: false. The filter will add the following HTTP headers when the backend service does not already provide the header:
        • StrictTransportSecurity: maxage=31536000; includeSubDomains
        • XFrameOptions: SAMEORIGIN
        • XContentTypeOptions: nosniff
        • ReferrerPolicy: strictorigin
      • In the event these additional headers cause issues with any deployment, you can disable the filter.

    • Nexus

      • Nexus realms configuration has been moved and is no longer nested under sso. The realm key has been renamed to realms, e.g.:
      addons:
        nexusRepositoryManager:
          values:
            realms:
              - "DockerToken"

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.32.6.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.8+rke2r1
    • K3s: v1.30.6+k3s1
    • EKS: v1.29.8
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.29.8-rke2r1-rocky-8-base-v1.1.1-stig-2024-09-23T08-14-20Z
    • EKS AMI: structsure-eks-1.29.8-rocky-8-base-v1.1.1-stig-2024-10-28T08-12-34Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

6.2.0 (2024-11-12)​

📦 SmoothGlue Features​

  • The SSO buttons default to read SmoothGlue SSO , when possible.

⏩ Upgraded Packages​

  • Upgrades to console v6.2.x (also v6.1.x) include the following:
    • Fixed bug in SG Run Basic where tools are missing from tools page
    • Fixed bug where deployments are not presented if they have not synced in Argo
    • Console teams now have a slug attribute
    • Bug fix: removing a user from an org now removes the user from the org's teams
    • Removed non-functional rename organization action
    • Bug fix: dashboard view does not crash if there is an error retrieving tool info
    • Removed ability to set user's organization attributes from team page
    • Increased click target for organization and project cards
  • This release of SmoothGlue Enterprise v6.2.0 includes Big Bang Version 2.39.0. For more details on the features and updates included in Big Bang Version 2.39.0, please refer to the Big Bang Release Notes.
  • nexus-iq chart upgraded to 183
  • Jira has been upgraded to 9.12.14:
    • Update gluon patch from 0.5.4 to 0.5.8
    • Update cypress (source) 13.15.0 -> 13.15.1

🐞 Bug Fixes​

  • Vault can now be configured to use the correct DNS suffix for the ISO regions.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.32.6.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.8+rke2r1
    • K3s: v1.30.5+k3s1
    • EKS: v1.29.6
  • The following AMI versions were used for testing:
    • RKE2 AMI: smoothglue-rke2-v1.29.8-rke2r1-rocky-8-base-v1.1.1-stig-2024-09-23T08-14-20Z
    • EKS AMI: smoothglue-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-09-09T08-14-46Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

6.1.0 (2024-10-30)​

The following are the v6.1.0 release notes for convenience:

🚨 Upgrade Notices​

This is a major update to SonarQube. During upgrade, you may get a SonarQube is under maintenance error message on the SonarQube UI.

  • To resolve this, once the HelmRelease upgrades, you will be prompted to visit your SonarQube instance at a <sonarqube_url>/setup URL. This is intended to launch a Database migration/update for SonarQube internally. The app will be available once it completes.

⏩ Upgraded Packages​

  • Console has been upgraded from v5.58 to v6.0.x and now offers the following capabilities:
    • Enhanced deployment wizard for deploying apps via Kustomize manifest (platform admins only).
    • Adds support for multiple ingress routes when using deployment wizard.
    • Adds ENABLE_SELF_SERVE_DEPLOYMENTS feature flag.
    • Fixes failure to load projects page if an expected deployment has no metadata in Argo CD.
    • Includes SonarQube, when it's deployed, on tools pages.
    • Removes non-functional rename project action.
    • Restores ability for platform admins to send credentials reset to users.
  • Big Bang has been upgraded from 2.37.0 to 2.38.0. For more details on the features and updates included in Big Bang Version 2.38.0, please refer to the Big Bang release notes.

🪲 Bug Fixes​

  • Fixes an issue with load balancer stickiness.
    • When compatibility_mode is set, the object that is returned contains false. Additionally, started deprecation for IaC variable name for EKS; the old variable will remain for the time being so please only set one:
      • var.sso_nlb_stickiness_enabled => var.sso_nlb_stickiness_settings
      • var.application_nlb_stickiness_enabled => var.application_nlb_stickiness_settings
  • Updates to the RKE2 Terraform to handle multiple VPC CIDRs.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.32.6.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.8+rke2r1
    • K3s: v1.30.5+k3s1
    • EKS: v1.29.6
  • The following AMI versions were used for testing:
    • RKE2 AMI: structsure-rke2-v1.29.8-rke2r1-rocky-8-base-v1.1.1-stig-2024-09-23T08-14-20Z
    • EKS AMI: structsure-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-09-30T08-10-58Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

6.0.1 (2024-10-22)​

Patch notes​

The following are the v6.0.0 release notes for convenience:

🚨 Upgrade Notices​

  • With the 6.0.0 release, Structsure Enterprise is now SmoothGlue Enterprise!
    • Structsure Enterprise Developer Collaboration Environment is now known as SmoothGlue Build Enterprise.
    • Structsure Enterprise Deploy Target is now known as SmoothGlue Run Enterprise.
  • On new installations making use of automatic Keycloak configuration, the default realm will now be named smoothglue instead of structsure. Keycloak should use redirect to this realm automatically, but any URL references to the structsure realm should be updated to point to the smoothglue realm on new installs.

    • For existing Structsure Enterprise installations making use of the automatic Keycloak realm configuration feature, you should add the following configuration to your bigbang-values.yaml in order to continue using your existing Keycloak realm:

      addons:
        keycloak:
          values:
            realms:
              - realmName: structsure

📦 SmoothGlue Enterprise Features​

  • The following applications have updated SmoothGlue branding/theming:
    • SmoothGlue Console
    • Keycloak
    • ArgoCD
  • External Secrets Operator: Now officially bundled and supported as a SmoothGlue Enterprise component.
    • External Secrets Operator (ESO) is not installed by default on either Build or Run installations and must be enabled explicitly.

      • ESO can be enabled by setting EXTERNAL_SECRETS_ENABLED to true in your Zarf config, or by adding the following settings to your bigbang-values.yaml:

        addons:
          externalSecrets:
            enabled: true
    • If you have previously manually installed External Secrets Operator, you may need to manually update Helm annotations to allow existing resources to be adopted.
  • IaC: The SmoothGlue Enterprise IaC now includes an optional Terragrunt module to create a public or private Route 53 zone associated with the cluster. Please see the documentation for more information on how to enable and configure this module.
  • IaC: SmoothGlue Enterprise now has provides an optional Terragrunt module to create an RDS database for Nexus IQ. Please see the documentation for more information on how to enable and configure this module.

⏩ Upgraded Packages​

  • Upgraded console to v5.57.x
  • This release includes Big Bang Version 2.37.0. For more details on the features and updates included in Big Bang Version 2.37.0, please refer to the Big Bang release notes.
  • Upgraded cert-manager to v1.15.3

🐞 Bug Fixes​

  • When migrating an existing Structsure Enterprise install to SmoothGlue Enterprise v6.0.0, a new Zarf Helm chart is deployed within the default namespace with an updated Crossplane Configuration for structsure-enterprise. The Helm chart fails to automatically patch the Configuration resource properly, even though it adopts the existing resource as expected. To work around this, a Zarf command will override the Configuration using a kubectl patch this release. This will not affect future releases.
  • Fixed an issue preventing user-provided TLS certificates using ZARF_VAR_CERT and ZARF_VAR_KEY from being consumed during the Zarf package deploy. If a user-provided certificate is provided, it will take precedence; otherwise, the full order of precedence is as follows, from highest to lowest:
    • certs/keys provided using ZARF_VAR_CERT or ZARF_VAR_KEY,
    • values provided to Istio using bigbang-secrets.yaml,
    • existing Istio secrets on the cluster, and finally
    • an automatically generated TLS certificate.
  • IaC: In VPCs with multiple CIDR ranges, the default security group rules for EKS clusters will now allow access from all CIDR ranges associated with the VPC, rather than just the primary CIDR block.

❗ Known Issues​

  • Although many visible references to Structsure Enterprise have been updated to SmoothGlue Enterprise, some resources will continue to use the older Structsure naming, largely for compatibility reasons. If any of the below listed resources still refer to Structsure, this is expected. These resources may be migrated to use the updated SmoothGlue branding in future releases.
    • When creating the smoothglue Keycloak realm, the default Keycloak configuration will continue to create groups for _structsureAdmins and _structsureAudit for use with Console.
    • Kubernetes objects which were previously deployed to the structsure-system namespace will continue to be deployed there, such as -overrides ConfigMaps and Secrets, as well as Crossplane and Flux resources.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.32.6.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.9-rke2r1
    • K3S: v1.30.5+k3s1
    • EKS: v1.29.8
  • The following AMI versions were used for testing:
    • RKE2 AMI: structsure-rke2-v1.29.9-rke2r1-rocky-8-base-v1.1.1-stig-2024-10-14T08-10-40Z
    • EKS AMI: structsure-eks-1.29.8-rocky-8-base-v1.1.1-stig-2024-10-14T08-10-55Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

6.0.0 (2024-10-16)​

🚨 Upgrade Notices​

  • With the 6.0.0 release, Structsure Enterprise is now SmoothGlue Enterprise!
    • Structsure Enterprise Developer Collaboration Environment is now known as SmoothGlue Build Enterprise.
    • Structsure Enterprise Deploy Target is now known as SmoothGlue Run Enterprise.
  • On new installations making use of automatic Keycloak configuration, the default realm will now be named smoothglue instead of structsure. Keycloak should use redirect to this realm automatically, but any URL references to the structsure realm should be updated to point to the smoothglue realm on new installs.

    • For existing Structsure Enterprise installations making use of the automatic Keycloak realm configuration feature, you should add the following configuration to your bigbang-values.yaml in order to continue using your existing Keycloak realm:

      addons:
        keycloak:
          values:
            realms:
              - realmName: structsure

📦 SmoothGlue Enterprise Features​

  • The following applications have updated SmoothGlue branding/theming:
    • SmoothGlue Console
    • Keycloak
    • ArgoCD
  • External Secrets Operator: Now officially bundled and supported as a SmoothGlue Enterprise component.
    • External Secrets Operator (ESO) is not installed by default on either Build or Run installations and must be enabled explicitly.

      • ESO can be enabled by setting EXTERNAL_SECRETS_ENABLED to true in your Zarf config, or by adding the following settings to your bigbang-values.yaml:

        addons:
          externalSecrets:
            enabled: true
    • If you have previously manually installed External Secrets Operator, you may need to manually update Helm annotations to allow existing resources to be adopted.
  • IaC: The SmoothGlue Enterprise IaC now includes an optional Terragrunt module to create a public or private Route 53 zone associated with the cluster. Please see the documentation for more information on how to enable and configure this module.
  • IaC: SmoothGlue Enterprise now has provides an optional Terragrunt module to create an RDS database for Nexus IQ. Please see the documentation for more information on how to enable and configure this module.

⏩ Upgraded Packages​

  • Upgraded console to v5.57.x
  • This release includes Big Bang Version 2.37.0. For more details on the features and updates included in Big Bang Version 2.37.0, please refer to the Big Bang release notes.
  • Upgraded cert-manager to v1.15.3

🐞 Bug Fixes​

  • When migrating an existing Structsure Enterprise install to SmoothGlue Enterprise v6.0.0, a new Zarf Helm chart is deployed within the default namespace with an updated Crossplane Configuration for structsure-enterprise. The Helm chart fails to automatically patch the Configuration resource properly, even though it adopts the existing resource as expected. To work around this, a Zarf command will override the Configuration using a kubectl patch this release. This will not affect future releases.
  • Fixed an issue preventing user-provided TLS certificates using ZARF_VAR_CERT and ZARF_VAR_KEY from being consumed during the Zarf package deploy. If a user-provided certificate is provided, it will take precedence; otherwise, the full order of precedence is as follows, from highest to lowest:
    • certs/keys provided using ZARF_VAR_CERT or ZARF_VAR_KEY,
    • values provided to Istio using bigbang-secrets.yaml,
    • existing Istio secrets on the cluster, and finally
    • an automatically generated TLS certificate.
  • IaC: In VPCs with multiple CIDR ranges, the default security group rules for EKS clusters will now allow access from all CIDR ranges associated with the VPC, rather than just the primary CIDR block.

❗ Known Issues​

  • Although many visible references to Structsure Enterprise have been updated to SmoothGlue Enterprise, some resources will continue to use the older Structsure naming, largely for compatibility reasons. If any of the below listed resources still refer to Structsure, this is expected. These resources may be migrated to use the updated SmoothGlue branding in future releases.
    • When creating the smoothglue Keycloak realm, the default Keycloak configuration will continue to create groups for _structsureAdmins and _structsureAudit for use with Console.
    • Kubernetes objects which were previously deployed to the structsure-system namespace will continue to be deployed there, such as -overrides ConfigMaps and Secrets, as well as Crossplane and Flux resources.

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.32.6.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.9-rke2r1
    • K3S: v1.30.5+k3s1
    • EKS: v1.29.8
  • The following AMI versions were used for testing:
    • RKE2 AMI: structsure-rke2-v1.29.9-rke2r1-rocky-8-base-v1.1.1-stig-2024-10-14T08-10-40Z
    • EKS AMI: structsure-eks-1.29.8-rocky-8-base-v1.1.1-stig-2024-10-14T08-10-55Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64