Setup Directory Structure

This document is intended to guide the System Integrator through the process of setting up a directory structure that will be used for installing / maintaining SmoothGlue environments. For the initial installation, this should exist on the bastion host.

In order to easily maintain multiple SmoothGlue environments, it is important to keep all relevant configuration for a SmoothGlue environment in a single place and to use a consistent pattern for that configuration.

The Install and Maintain documentation will reference the following example directory structure for a SmoothGlue environment:

example-org-run
├── .env
├── .terragrunt-cache
    └── providers
├── bigbang-secrets.yaml
├── bigbang-values.yaml
├── compiled
├── compile-config.sh
├── env.hcl
├── infra-iac
├── zarf-package-smoothglue-*.tar.zst
├── smoothglue-*-docs.tar.gz
├── smoothglue-*-docs-md.tar.gz
└── zarf-config.yaml

tip

Consider maintaining the directory in Git. Ensure secrets are encrypted when they are stored in Git.

Directory

The directory name should describe a unique environment. For example, example-org-run, would describe a run environment for the example-org organization.

Create the directory to store the configuration in:

# Following the guidance, provide a name for your environment's directory
ENVIRONMENT_NAME=example-org-run
mkdir $ENVIRONMENT_NAME

SmoothGlue Artifacts

After Getting Access to SmoothGlue Artifacts, download terraform_bundle and zarf-package-smoothglue for the desired SmoothGlue version from the appropriate S3 bucket for your AWS partition to your Bastion Host.

For example, you can download them from a commercial AWS account as follows:

# Update the SMOOTHGLUE_VERSION value based on available versions in the bucket
SMOOTHGLUE_VERSION=v6.10.0
aws s3 cp s3://structsure-releases/$SMOOTHGLUE_VERSION/terraform_bundle-$SMOOTHGLUE_VERSION.tar.gz $ENVIRONMENT_NAME/
aws s3 cp s3://structsure-releases/$SMOOTHGLUE_VERSION/zarf-package-smoothglue-amd64-$SMOOTHGLUE_VERSION.tar.zst $ENVIRONMENT_NAME/
aws s3 cp s3://structsure-releases/$SMOOTHGLUE_VERSION/smoothglue-$SMOOTHGLUE_VERSION-docs-md.tar.gz $ENVIRONMENT_NAME/
aws s3 cp s3://structsure-releases/$SMOOTHGLUE_VERSION/smoothglue-$SMOOTHGLUE_VERSION-docs.tar.gz $ENVIRONMENT_NAME/

The terraform_bundle is a compressed archive of the IaC. To extract the contents:

tar -xf $ENVIRONMENT_NAME/terraform_bundle-$SMOOTHGLUE_VERSION.tar.gz -C $ENVIRONMENT_NAME && rm $ENVIRONMENT_NAME/terraform_bundle-$SMOOTHGLUE_VERSION.tar.gz

There should now be a infra-iac directory with all of the IaC and base configuration files inside of example-org-run.

.terragrunt-cache/providers

The .terragrunt-cache/providers directory will hold all of the cached providers for the Structsure IaC. The main benefit of this file will be using a shared cache of Terraform providers across the modules included in the SmoothGlue IaC.

note

For the purposes of this example, the .terragrunt-cache/providers directory is in the example-org-run directory, but consider creating it in a location that could be shared by multiple SmoothGlue deployments.

Create the .terragrunt-cache/providers directory:

mkdir -p $ENVIRONMENT_NAME/.terragrunt-cache/providers/

Big Bang Config

The bigbang-values.yaml and bigbang-secrets.yaml files hold the user-defined configuration for Big Bang. These files are meant to be used to override default configuration provided SmoothGlue and Big Bang itself. These files are meant to be merged with configuration files generated from the SmoothGlue IaC, so please create these files regardless if they contain any configuration.

info

See How to Configure Big Bang Values for more information.

Create Big Bang config files:

touch $ENVIRONMENT_NAME/bigbang-values.yaml
touch $ENVIRONMENT_NAME/bigbang-secrets.yaml

zarf-config.yaml

The zarf-config.yaml file will provide configuration for the SmoothGlue package. Please create the zarf-config.yaml file under example-org-run with the following minimum config that needs to be set by the System Integrator:

package:
  deploy:
    timeout: 30m0s
    set:
      CLUSTER_TYPE: run
      DOMAIN: example.com
      # Removing the lines containing CERT_PATH, KEY_PATH, and CA_CERT_PATH will
      #  generate self-signed certificates for the configured DOMAIN.
      CERT_PATH: /path/to/server-cert.pem
      KEY_PATH: /path/to/server-key.pem
      CA_CERT_PATH: /path/to/ca-cert.pem
      
      # Do not modify the BIGBANG_* lines.
      BIGBANG_VALUES_FILE: compiled/bigbang-values.yaml
      BIGBANG_SECRETS_FILE: compiled/bigbang-secrets.yaml

info

Please see How To Configure SmoothGlue Package for more information on configuring the SmoothGlue package.

compiled

The compiled directory will hold the dynamically generated configuration files mentioned above. These files are not meant to be edited directly. Instead, edit the user-provided config files.

Create the compiled directory:

mkdir $ENVIRONMENT_NAME/compiled

env.hcl

The env.hcl file holds the configuration values for the SmoothGlue IaC. The SmoothGlue IaC comes with an example env-run.hcl and env-build.hcl config files located under infra-iac/envs. When configuring a new cluster, the System Integrator should create a new env.hcl file by copying one of the example configurations and start modifying the values therein.

For example, copy the env-run.hcl file:

cp $ENVIRONMENT_NAME/infra-iac/envs/env-run.hcl $ENVIRONMENT_NAME/env.hcl

.env

The .env file stores all of the relevant environment variables for maintaining a SmoothGlue environment. When installing or maintaining the SmoothGlue environment, the System Integrator should first source .env in their terminal from within the example-org-run directory to load all of environment vairables. Create the .env file with the following contents:

# Set the Terraform plugin cache directory to avoid installing providers multiple times.
export TF_PLUGIN_CACHE_DIR="$(pwd)/.terragrunt-cache/providers/"

# Set the path to the SmoothGlue IaC to execute.
export TG_WORKING_DIR="$(pwd)/infra-iac/"

# Specify the environment configuration file for SmoothGlue Run deployment.
export TERRAGRUNT_ENV_FILE="$(pwd)/env.hcl"

# Define a unique Terraform workspace name for this deployment. All cloud resources will be created under this workspace.
export WORKSPACE_NAME="example-org-run"

# Specify the Kubernetes distribution; in this case, EKS.
export K8S_DISTRO="eks-cluster"

# Define the type of SmoothGlue deployment.
export CLUSTER_TYPE="run"

# Path to the compiled zarf-config.yaml file
export ZARF_CONFIG="$(pwd)/compiled/zarf-config.yaml"

tip

Consider setting the Terraform workspace name to same name as the directory. The SmoothGlue IaC will prepend most cloud resources with the workspace name to make managed cloud resources easily identifiable from the AWS console

compile-config.sh

Applications often need databases and object storage to store data. The SmoothGlue IaC will create the appropriate storage type in AWS and generates some configuration based on the cloud resources it creates. Therefore, to layer user-provided config ontop of the SmoothGlue generated config, the following script will dynamically generate a compiled version. Please create a file named compile-config.sh with the following contents:

SMOOTHGLUE_IAC_PATH="$(pwd)/infra-iac/"

# Compile BigBang secret files
zarf tools yq ea '. as $i ireduce ({}; . * $i)' \
${SMOOTHGLUE_IAC_PATH}/files/bigbang-secrets.yaml \
$(find ${SMOOTHGLUE_IAC_PATH}/outputs/ -name 'bigbang-secrets-*.yaml') \
bigbang-secrets.yaml > compiled/bigbang-secrets.yaml

# Compile BigBang value files
zarf tools yq ea '. as $i ireduce ({}; . * $i)' \
${SMOOTHGLUE_IAC_PATH}/files/bigbang-values.yaml \
$(find ${SMOOTHGLUE_IAC_PATH}/outputs/ -name 'bigbang-values-*.yaml') \
bigbang-values.yaml > compiled/bigbang-values.yaml

# Compile zarf-config files
zarf tools yq ea '. as $i ireduce ({}; . * $i)' \
$(find ${SMOOTHGLUE_IAC_PATH}/outputs/ -name 'zarf-config-*.yaml') \
zarf-config.yaml > compiled/zarf-config.yaml

Ensure that the script is executable:

chmod +x $ENVIRONMENT_NAME/compile-config.sh

Generate Script

The following script will create all of described content above:

# Update the following
export ORG_NAME="example-org"
export CLUSTER_TYPE="run"
export SMOOTHGLUE_VERSION="v6.10.0"

# Begin Script
export DIR="$ORG_NAME-$CLUSTER_TYPE"

# Create Directories
mkdir $DIR
mkdir -p $DIR/.terragrunt-cache/providers/
mkdir $DIR/compiled

# Create Big Bang Config Files
touch $DIR/bigbang-values.yaml
touch $DIR/bigbang-secrets.yaml

# Download SmoothGlue Artifacts
aws s3 cp s3://structsure-releases/$SMOOTHGLUE_VERSION/terraform_bundle-$SMOOTHGLUE_VERSION.tar.gz $DIR/
aws s3 cp s3://structsure-releases/$SMOOTHGLUE_VERSION/zarf-package-smoothglue-amd64-$SMOOTHGLUE_VERSION.tar.zst $DIR/

# Extract SmoothGlue IaC Bundle
tar -xf $DIR/terraform_bundle-$SMOOTHGLUE_VERSION.tar.gz -C $DIR && rm $DIR/terraform_bundle-$SMOOTHGLUE_VERSION.tar.gz

# Copy IaC Default Config for CLUSTER_TYPE
cp $DIR/infra-iac/envs/env-$CLUSTER_TYPE.hcl $DIR/env.hcl

# Generate zarf-config.yaml
cat << EOF > $DIR/zarf-config.yaml
package:
  deploy:
    timeout: 30m0s
    set:
      CLUSTER_TYPE: $CLUSTER_TYPE
      DOMAIN: example.com
      CERT_PATH: /path/to/server-cert.pem
      KEY_PATH: /path/to/server-key.pem
      CA_CERT_PATH: /path/to/ca-cert.pem
      
      # These can be left alone
      BIGBANG_VALUES_FILE: compiled/bigbang-values.yaml
      BIGBANG_SECRETS_FILE: compiled/bigbang-secrets.yaml
EOF

# Generate .env
cat << EOF > $DIR/.env
# Set the Terraform plugin cache directory to avoid installing providers multiple times.
export TF_PLUGIN_CACHE_DIR="\$(pwd)/.terragrunt-cache/providers/"

# Set the path to the SmoothGlue IaC to execute.
export TG_WORKING_DIR="\$(pwd)/infra-iac/"

# Specify the environment configuration file for SmoothGlue Run deployment.
export TERRAGRUNT_ENV_FILE="\$(pwd)/env.hcl"

# Define a unique Terraform workspace name for this deployment. All cloud resources will be created under this workspace.
export WORKSPACE_NAME="$DIR"

# Specify the Kubernetes distribution; in this case, EKS.
export K8S_DISTRO="eks-cluster"

# Define the type of SmoothGlue deployment.
export CLUSTER_TYPE="$CLUSTER_TYPE"

# Path to the compiled zarf-config.yaml file
export ZARF_CONFIG="\$(pwd)/compiled/zarf-config.yaml"
EOF

# Generate compile-config.sh
cat << EOF > $DIR/compile-config.sh
SMOOTHGLUE_IAC_PATH="\$(pwd)/infra-iac/"

# Compile BigBang secret files
zarf tools yq ea '. as \$i ireduce ({}; . * \$i)' \
\${SMOOTHGLUE_IAC_PATH}/files/bigbang-secrets.yaml \
\$(find \${SMOOTHGLUE_IAC_PATH}/outputs/ -name 'bigbang-secrets-*.yaml') \
bigbang-secrets.yaml > compiled/bigbang-secrets.yaml

# Compile BigBang value files
zarf tools yq ea '. as \$i ireduce ({}; . * \$i)' \
\${SMOOTHGLUE_IAC_PATH}/files/bigbang-values.yaml \
\$(find \${SMOOTHGLUE_IAC_PATH}/outputs/ -name 'bigbang-values-*.yaml') \
bigbang-values.yaml > compiled/bigbang-values.yaml

# Compile zarf-config files
zarf tools yq ea '. as \$i ireduce ({}; . * \$i)' \
\$(find \${SMOOTHGLUE_IAC_PATH}/outputs/ -name 'zarf-config-*.yaml') \
zarf-config.yaml > compiled/zarf-config.yaml
EOF

# Ensure compile-config.sh is executable
chmod +x $DIR/compile-config.sh