Skip to main content
Version: 5.21.0

Release Notes

5.21.0 (2024-10-01)​

📦 Structsure Features​

  • Vault can now be configured with a CA cert when talking to AWS services like KMS. This is relevant when Vault is configured to use KMS in higher environments to auto unseal and can now be set via the CA_CERT_AWS configuration option in the Zarf Config file as explained in the "Install -> Installation Options" section of the Structsure documentation.

⏩ Upgraded Packages​

  • Jira has been upgraded from 9.12.12 to 9.12.13
  • Console has been upgraded from v5.55 to v5.56
    • Fixed flickering page heading text
    • Improved misleading and confusing log messages
    • Updated Next.js dependencies to address new vulnerability
    • Set Argo CD project permissions when creating deployments
  • Big Bang has been upgraded from 2.35.0 to 2.36.0. For more details on the features and updates included in Big Bang Version 2.36.0, please refer to the Big Bang release notes

🪲 Bug Fixes​

  • Fixed an issue that prevented nip.io certs from being disabled when trying to use cert-manager. Documentation on how to properly enable cert-manager can be found in the Structsure documentation under "How-To Guides > Applications > Cert-Manager > How to setup and install cert-manager for structsure"

❗️ Known Issues​

  • Incorrect Virtual Service Host Configuration in Loki Scalable Mode

🌐 Compatibility​

  • The packages for this release were built using Zarf v0.32.6.
  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.8+rke2r1
    • K3S: v1.30.5+k3s1
    • EKS: v1.29.6
  • The following AMI versions were used for testing:
    • RKE2 AMI: structsure-rke2-v1.29.8-rke2r1-rocky-8-base-v1.1.1-stig-2024-09-23T08-14-20Z
    • EKS AMI: structsure-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-09-30T08-10-58Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

🔗 Helpful Links​

  • Refer to the Structsure documentation for additional guidance.
  • For details on the Big Bang release, see the Big Bang Release Notes.

5.20.0 (2024-09-23)​

🎉 This release of Structsure Enterprise v5.20.0 introduces several important updates, including Big Bang Version 2.35.0. For detailed information on the new features and updates included in Big Bang Version 2.35.0, please refer to the Big Bang release notes.

🔧 Upgrade Notices​

🚨 Big Bang Upgrade

  • Istio-controlplane - MR:
    • Istio gets updated to 1.22.4. Big Bang apps should automatically cycle to get the latest sidecar version and config. Be sure to cycle pods for any community or tenant applications manually.
  • Mattermost - MR:
    • Postgresql using the builtin bitnami module does not upgrade gracefully. You must manually backup and restore your database before accepting this upgrade. Using the builtin postgresql module is not a supported configuration in production environments. If you are using the IaC module for Mattermost, this warning can be ignored.
  • Automated the Velero temporary manual fix suggested by Big Bang. You don’t need to apply the temporary fix mentioned in the third bullet point yourself: https://repo1.dso.mil/big-bang/bigbang/-/releases/2.35.0#upgrade-notices

📢 Console Upgrade to v5.55.x

  • Expand reliance on database tool records instead of env vars
  • Use more reliable metadata to discover tools via virtual service queries
  • Initialize tool records by querying virtual services instead of env vars
  • Present extended platform tools in non-AMI use cases

📢 Dynamic dbconfig.xml for Jira and confluence

  • Enabled dynamic creation of dbconfig.xml for Jira and Confluence.
  • Set forceConfigUpdate to true by default, ensuring the dbconfig.xml is recreated on every pod restart.
  • Users can override this behavior by setting forceConfigUpdate to false in their claim to preserve the file across restarts.

📢 Enhance eks-cluster Terraform Module

  • Clarified that s3_kms_key_id must be provided as a full ARN, not just the key name or ID.
  • Made object ownership configurable between "BucketOwnerPreferred" and "BucketOwnerEnforced."
  • Enforced secure S3 bucket creation with attach_deny_insecure_transport_policy = true.
  • Added validation for s3_object_ownership to ensure valid values like "BucketOwnerPreferred" or "BucketOwnerEnforced."

🪲 Bigbang HelmRelease Reconciles Before Zarf Deploy Finishes

  • Starting with this release, the bigbang Flux HelmRelease object in the bigbang namespace will be temporarily suspended when starting the upgrade package deployment. If the bigbang HelmRelease is suspended outside of this automatic suspension, it will trigger a failure of the upgrade package deployment starting with this release, since suspension of the HelmRelease object will prevent the upgrade from completing successfully.

🪲 Remove Temporary Neuvector Command

  • Removed a pin that deleted crds nvvulnerabilityprofiles.neuvector.com and nvcomplianceprofiles.neuvector.com at zarf start. No longer required after Big Bang 2.7.6-bb.0

🪲 Vault-Agent Trust CAs

  • Fixed an issue that prevented vault-agents from trusting Certificate Authorities when providing a CA cert as an input to the Zarf package. This update re-enables a Bigbang-managed integration between Prometheus and Vault. If you have deployed Vault prior to this update, ensure you followed the documented initialization instructions for Vault including the steps for configuring the Vault policies for Prometheus. Prometheus will be offline until Vault has been configured.

🪲 Force Mattermost to use Database Connection String

  • Fixed an issue where Mattermost wouldn't respect when database connection string changed. Database password and hostname can now be changed for Mattermost.

🪲 Upgrade Gitlab to v17.2.7 to Resolve Critical CVE

🧩 Zarf Version​

  • The packages for this release were built using Zarf v0.32.6.

🌐 Kubernetes Distributions and Versions​

  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.8+rke2r1
    • K3S: v1.30.5+k3s1
    • EKS: v1.29.6

📦 AMI Versions​

  • The following AMI versions were used for testing:
    • RKE2 AMI: structsure-rke2-v1.29.8-rke2r1-rocky-8-base-v1.1.1-stig-2024-09-23T08-14-20Z
    • EKS AMI: structsure-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-09-09T08-14-46Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

📝 Changelog​

:tools:️ Infrastructure as Code (IaC) Features​

  • iac: enhance eks-cluster terraform module

📦 Package Features​

  • bump console to v5.55.x
  • helm: dynamic dbconfig.xml for Jira & confluence
  • upgrade BB to 2.35

🪲 Bug Fixes​

  • bigbang HelmRelease reconciles before zarf deploy finishes
  • force mattermost to use database connection string
  • remove temp neuvector cmd
  • upgrade gitlab to v17.2.7 to resolve critical CVE
  • vault-agent trust CAs

🛑 Known Issues​

  • ❗ Incorrect Virtual Service Host Configuration in Loki Scalable Mode

🔗 Helpful Links​

  • Refer to the Structsure documentation for additional guidance.
  • For details on the Big Bang release, see the Big Bang Release Notes.

5.19.0 (2024-09-04)​

🎉 This release of Structsure Enterprise v5.19.0 introduces several important updates, including Big Bang Version 2.34.0. For detailed information on the new features and updates included in Big Bang Version 2.34.0, please refer to the Big Bang release notes.

🔧 Upgrade Notices​

🚨 Big Bang Upgrade​

  • Nexus:

    • ⚠️ Breaking Changes:
      • Nexus 3.71.0-06 removes support for internal OrientDB and replaces it with H2.
      • Nexus 3.71.0-06 requires Java 17+ (previously supported Java 8 and 11).
      • ⚠️ Migration Required: If you are using an internal database, refer to the migration steps before upgrading.

  • Minio-operator:

    • The MinIO Operator Console has been deprecated and removed starting from version 6.0.0.

  • BigBang:

    • Resolved an issue with an invalid value in the images.txt release artifact.

📢 Confluence Upgrade​

  • This release includes a major version upgrade of Confluence from 8.9.4 to 9.0.2.
  • For detailed upgrade notes from Atlassian, refer to the Confluence 9.0 upgrade notes.

🚨 Important: Please update the miniOrange SSO app and any other apps you are using to ensure compatibility with Confluence version 9.0.

✨ Major Features​

🛠️ Containerd Iron Bank Mirror​

  • Structsure Enterprise now includes a built-in containerd mirror for mirroring Iron Bank images to Zarf's internal registry. This mirror is enabled by default, and instructions on how to disable it are available here.

🔍 Compatibility​

🧩 Zarf Version​

  • The packages for this release were built using Zarf v0.32.6.

🌐 Kubernetes Distributions and Versions​

  • The packages were tested across the following Kubernetes distributions:
    • RKE2: v1.29.7+rke2r1
    • K3S: v1.30.0
    • EKS: v1.29

📦 AMI Versions​

  • The following AMI versions were used for testing:
    • RKE2 AMI: Structsure-rke2-v1.29.7-rke2r1-rocky-8-base-v1.1.1-stig-2024-08-12T08-14-46Z
    • EKS AMI: Structsure-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-07-29T08-12-23Z
    • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

📝 Changelog​

🛠️ Infrastructure as Code (IaC) Features​

  • iac: Added toggle for containerd Iron Bank mirror.
  • iac: Changed rds_engine_version type to string.

📦 Package Features​

  • Enabled Loki by default in DT and collab.
  • Updated console to version 5.54.x.
  • Upgraded Big Bang to version 2.34.0.
  • Upgraded third-party Big Bang apps.

🐞 Bug Fixes​

  • Fixed Kyverno policy exclusion and S3 region endpoint for GitLab backup.
  • Restored broken cluster autoscaler functionality.
  • Resolved cluster autoscaler issue.

📚 Documentation Updates​

  • Updated Neuvector upgrade documentation.
  • Added RKE2 IaC reference documentation.
  • Corrected spelling errors.
  • Updated console documentation.

🛑 Known Issues​

  • ❗ Incorrect Virtual Service Host Configuration in Loki Scalable Mode

🔗 Helpful Links​

  • Refer to the Structsure documentation for additional guidance.
  • For details on the Big Bang release, see the Big Bang Release Notes.

5.18.0 (2024-08-21)​

This release of Structsure Enterprise v5.18.0 includes Big Bang Version 2.33.0. For more details on the features and updates included in Big Bang Version 2.33.0, please refer to the Big Bang release notes.

Upgrade Notices​

EKS Default Node Group Naming​

The eks-cluster IaC module now supports a variable called default_eks_node_group_name, which allows specifying the name for the default node group and its EC2 instances. If the value is empty, the default node group will inherit the cluster's name.

If the node group name is changed, this will trigger a node group replacement. To avoid triggering node group replacement inadvertently on existing clusters, if this value is not set, the default will match the previous value, "structsure-nodes". If the default node group is disabled using the disable_default_node_groups variable (usually used in conjunction with additional_eks_managed_groups), the default_eks_node_group_name variable will have no effect.

Compatibility​

Zarf Version​

The packages for this release were built using the following Zarf version:

  • Zarf: v0.32.6

Kubernetes Distributions and Versions​

The packages were tested across the following Kubernetes distributions and versions:

  • Rancher Kubernetes Engine 2 (RKE2): v1.29.7+rke2r1
  • Kubernetes Lightweight (K3S): v1.30.0
  • Elastic Kubernetes Service (EKS): v1.29

AMI Versions​

The following AMI versions were used for testing:

  • RKE2 AMI: Structsure-rke2-v1.29.7-rke2r1-rocky-8-base-v1.1.1-stig-2024-08-12T08-14-46Z
  • EKS AMI: Structsure-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-07-29T08-12-23Z
  • Base AMI: Rocky-8-EC2-LVM-8.10-20240528.0.x86_64

Changelog​

IaC Features​

  • Iac: Allow configuration of the RDS backup retention period
  • Iac: Allow passing root CAs as strings or base64
  • Iac: EKS default node group name supports inherited cluster name

Package Features​

  • Upgrade Big Bang to v2.33.0

Package Bug Fixes​

  • Collect existing Big Bang values and merge into provided values
  • Default values prevent Console deploy
  • Handle empty existing bigbang-overrides
  • Prevent failed upgrades from creating multiple XRs

Documentation​

  • Explicitly export zarf_config variable in upgrade documentation
  • Check out the documentation for guidance
  • Big Bang v2.33.0 Release Notes

5.17.1 (2024-08-26)​

Package Bug Fixes​

  • collect existing bigbang values and merge into provided values
  • default values prevent console deploy
  • handle empty existing bigbang-overrides
  • prevent failed upgrades from creating multiple XRs

5.17.0 (2024-08-16)​

This release of Structsure Enterprise v5.17.0 includes Big Bang Version 2.32.0. For more details on the features and updates included in Big Bang Version 2.32.0, please refer to the Big Bang release notes.

Upgrade Notices​

Kubernetes Upgrade Requirement​

This release requires Kubernetes native sidecars. Hence, an upgrade to Kubernetes v1.29 or later is required.

Post-Upgrade Instructions​

To ensure the istio-proxy sidecar container switches to an init container (due to Kubernetes native sidecar support), restart all Istio-supporting pods immediately after the upgrade. Use the following commands:

for ns in $(kubectl get ns -l app.kubernetes.io/part-of=bigbang,istio-injection=enabled -o custom-columns=":metadata.name"); do
  kubectl rollout restart deployment -n $ns
  kubectl rollout restart statefulset -n $ns
  kubectl rollout restart daemonset -n $ns
done

Backup Recommendations​

It is highly advised to create backups of current admin passwords before performing the Structsure upgrade for the following applications to prevent the loss of credentials:

  • Grafana
  • Keycloak
  • SonarQube
  • NeuVector

Default Admin Passwords​

If not changed manually, the default admin passwords are as follows:

  • Grafana: Monitoring-grafana secret within the monitoring namespace. This password will be overridden to a randomly generated strong password.
  • Keycloak: keycloak-env secret within the keycloak namespace. In tested upgrades, the admin password didn’t change when only the composition was applied, but it may change during the full upgrade with the Structsure Zarf package due to changes in the default chart values.
  • SonarQube: Default username and password are both admin. The password is prompted to change upon first login. Similar to Keycloak, the password may change during the full upgrade.
  • NeuVector: Default username and password are both admin. The admin and metrics users' passwords will be overridden to randomly generated strong passwords.

Major Features​

Persistent Volumes for Monitoring Applications​

We now offer persistence via Persistent Volume Claims (PVCs) for the following monitoring applications:

  • Grafana
  • Prometheus
  • Alertmanager

Enabling Persistence​

To enable persistence, set persistence = true in the cluster_inputs object in your hcl and rerun your Terragrunt stage. Caution: If you are currently overriding, providing custom PVCs, or using a third-party solution for these applications, your settings might be overwritten.

Specific Considerations​

  • Grafana: When persistence is enabled, only one pod is allowed due to the ReadWriteOnce access mode of the default EBS storage class. Refer to the docs to configure ReadWriteMany (e.g., Amazon EFS) if multiple pods are desired.
  • Prometheus: Allows for adjusting the retention period and retention file size.

Compatibility​

Zarf Version​

The packages for this release were built using the following Zarf version:

  • Zarf: v0.32.6

Kubernetes Distributions and Versions​

The packages were tested across the following Kubernetes distributions and versions:

  • Rancher Kubernetes Engine 2 (RKE2): v1.29.7+rke2r1
  • Kubernetes Lightweight (K3S): v1.30.0
  • Elastic Kubernetes Service (EKS): v1.29

AMI Versions​

The following AMI versions were used for testing:

  • RKE2 AMI: structsure-rke2-v1.29.7-rke2r1-rocky-8-base-v1.1.1-stig-2024-08-12T08-14-46Z
  • EKS AMI: structsure-eks-1.29.6-rocky-8-base-v1.1.1-stig-2024-07-29T08-12-23Z
  • Base AMI: rocky-8-EC2-LVM-8.10-20240528.0.x86_64

Changelog​

IaC Bug Fixes​

  • Iac: Always set S3 regionendpoint for Zarf init
  • Iac: Automatically remove duplicates in allowed_security_groups

Package Features​

  • Bump Console version to 5.52.x
  • Crossplane: Argo CD declarative SSO
  • Crossplane: NeuVector is configured with read-only Prometheus user and admin user with random passwords
  • Crossplane: Web app default passwords randomly generated
  • Increment RKE2 K8s to 1.29, fail Zarf package deploy if K8s version < 1.29
  • Persistent storage for monitoring apps
  • Upgrade Big Bang to v2.32.0

Package Bug Fixes​

  • Dig Keycloak config realm to avoid nil pointer during Zarf deploy
  • Prevent Kyverno policy from erroneously Helm templating values
  • Revert rendering of values in Structsure-enterprise chart
  • Use appropriate whitespace in generate Kyverno policy

Documentation​

  • Updated Zarf version in documentation
  • Check out the documentation for guidance
  • Big Bang v2.32.0 Release Notes

5.16.1 (2024-08-26)​

Package Bug Fixes​

  • collect existing bigbang values and merge into provided values
  • dig keycloak config realm to avoid nil pointer during zarf deploy
  • handle empty existing bigbang-overrides
  • prevent failed upgrades from creating multiple XRs
  • prevent kyverno policy from erroneously helm templating values
  • revert rendering of values in structsure-enterprise chart
  • use appropriate whitespace in generate kyverno policy

5.16.0 (2024-07-25)​

Warning​

  • Included in this update is a fix for crossplane pods to be excluded from a kyverno policy that was blocking scheduling for crossplane pods. The fix will be applied automatically as part of the package. The IaC creates some values for kyverno policies that are no longer needed as those have been moved to the package. If you upgrading from a previous version, you will need to review the kyverno-policies-overrides config map in the structsure-system namespace and remove any entries that the IaC added. IaC specific entries can be identified by viewing the value file outputs of the IaC.
  • Included in this update is better support for a containerd mirror of registry1.dso.mil to the internal zarf registry. Running the IaC to apply these changes will generate a new launch template and will cycle EKS nodes onto the new config. If you are using a Structsure EKS AMI, you will need to use an AMI that was built on or before 07/11/2024. If using an AMI prior to that build date, the containerd mirror will stop functioning until a new AMI is used.

IaC Bug Fixes​

  • iac: eks registry mirror for all environments

Package Features​

  • enable NetworkPolicy in vpc cni by default
  • Upgrade Big Bang to v2.31

Documentation​

  • don't render wiki/style guide as part of public docs site
  • Big Bang v2.31.0 Release Notes

5.15.1 (2024-08-26)​

Package Bug Fixes​

  • collect existing bigbang values and merge into provided values
  • dig keycloak config realm to avoid nil pointer during zarf deploy
  • handle empty existing bigbang-overrides
  • prevent failed upgrades from creating multiple XRs
  • prevent helm rollback from deleteing claims
  • prevent kyverno policy from erroneously helm templating values
  • revert rendering of values in structsure-enterprise chart
  • use appropriate whitespace in generate kyverno policy

5.15.0 (2024-07-11)​

Warning​

Changes to Console password generation require Terraform to generate a new password for pre-existing deployments. As a result, Console will be down from the time the IaC is ran until the cluster has reconciled the outputted BigBang values files.

Additional Notes​

As part of the Big Bang upgrade, Keycloak may have some trouble reconciling. We have tested that a way to ensure a smooth upgrade is to, before upgrading, delete the statefulsets for Keycloak in your cluster and let the new version upgrade the helm to bring it back. For more notes on this, please see the official documentation for Big Bang 2.30.

Kiali is now enabled by default.

The default storage location for SSH keys for cluster management have been moved from AWS Parameter Store to AWS Secrets Manager. As a result, the following IAM permissions are required in order to run the IAC: 

{ "Effect": "Allow", "Action": [ "secretsmanager:CreateSecret", "secretsmanager:TagResource" ], "Resource": "*" }

iac: rds_engine_version as var Users can now set the versions of each modules RDS via {tool}_inputs, such as confluence_inputs, in the hcl file.

IaC Features​

  • iac: EKS containerd mirror for ironbank images
    • For now, this is supported for Structsure AMIs. We have noticed that non-Structsure AMIs have an issue with the config and we are working on a follow-up solve.
  • iac: eks node group sizes as obj var
    • Users can now set the sizes (min, max, desired) of the default EKS nodegroup via hcl file.
  • iac: rds engine_version as var
    • Users can now set the versions of each modules RDS via {tool}-inputs in the hcl file. Note that users who have their Confluence RDS database version greater than 13.8 will revive an error upgrading infra until this release when they can set the RDS database version.
  • iac: Save SSH private keys to AWS Parameter store from SSM
    • The SSH key will now be found in the AWS Secrets Manager instead of the SSM Parameter store.
  • iac: update default iac k8s version to 1.28

IaC Bug Fixes​

  • iac: special characters within console database password

Package Features​

  • crossplane: automatic kiali sso integration
  • Enable docs search
  • Upgrade Big Bang to 2.30.0

Package Bug Fixes​

  • crossplane: crossplane exempt from kyverno drop all policy

Documentation​

  • new section on High Availability Configuration
  • Big Bang v2.30.0 Release Notes

5.14.2 (2024-08-26)​

Package Bug Fixes​

  • collect existing bigbang values and merge into provided values
  • crossplane: crossplane exempt from kyverno drop all policy
  • dig keycloak config realm to avoid nil pointer during zarf deploy
  • handle empty existing bigbang-overrides
  • prevent failed upgrades from creating multiple XRs
  • prevent helm rollback from deleteing claims
  • prevent kyverno policy from erroneously helm templating values
  • revert rendering of values in structsure-enterprise chart
  • use appropriate whitespace in generate kyverno policy

5.13.1 (2024-08-26)​

Package Bug Fixes​

  • collect existing bigbang values and merge into provided values
  • handle empty existing bigbang-overrides
  • prevent failed upgrades from creating multiple XRs
  • prevent helm rollback from deleteing claims
  • prevent kyverno policy from erroneously helm templating values
  • revert rendering of values in structsure-enterprise chart
  • use appropriate whitespace in generate kyverno policy