Create an RKE2 Cluster
Structsure's AWS IAC uses Terragrunt to provision its infrastructure in an automated, repeatable fashion. Terragrunt is a wrapper around the popular Terraform tool, which provides some quality of life improvements relating to building infrastructure using multiple interrelated Terraform modules.
Requirements
The IaC should be deployed on a Linux-based EC2 instance or container with
the following tools present in the PATH
:
- Git
- Terraform
- Terragrunt
These tools are provided as part of the Structsure Utility Image.
Clone the Repository
Start by cloning the Structsure AWS IAC Git repository. Note that this
repository currently makes use of Git submodules, so the basic git clone
command must be modified to also clone the required submodules:
git clone --recursive git@gitlab.com:structsure/breakerbar/structsure-aws-iac.git
To update or re-initialize the submodules on an already-cloned repository, run this command:
git submodule update --init
Git submodules are referenced using relative paths in the AWS IaC repository, so
if using an offline Git mirror, the rke2-aws-tf
Git repository should be at
the same folder level as the structsure-aws-iac
repository. Refer to the
.gitmodules
file for the specific configuration of each submodule.
Additionally, when using a local Git mirror, Git may need to be configured to allow cloning submodules from file. This can be done by running:
git config --global protocol.file.allow always
Configure Cluster Options
The behavior of Terraform can be modified by providing different variables to
Terraform modules and inputs to Terragrunt. The AWS IaC is set up so that most
of the required environment-specific configuration changes can be made in a
single env.hcl
file. Example configuration files for collaboration
environments and deploy targets can each be found in the infra-iac/envs/
directory
of the AWS IaC repository. When configuring a new cluster, make a new custom
env.hcl
file by copying one of the example configurations and start modifying
the values therein. For more information on the differences between
collaboration environments and deploy targets, see the Structsure
Overview page.
A custom env.hcl
file can be provided to Terragrunt by setting the
TERRAGRUNT_ENV_FILE
environment variable. Note that if a relative path is
used, it must be the relative path from each Terragrunt module, so for best
results, an absolute path should be used. If the TERRAGRUNT_ENV_FILE
environment variable is not set, then Terragrunt will look recursively through
the module's parent directories for a file called env-default.hcl
and source
it. In the IaC repository, this is a symlink located at
infra-iac/env-default.hcl
and may be re-pointed to the desired environment file.
The locals and inputs within the env.hcl
file are sourced in each Terragrunt
module's terragrunt.hcl
file. When a variable under locals
is also within inputs
, it is passed on to all Terraform modules as an input variable (e.g., compatibility_mode = false
set within locals
section of the env.hcl
will be passed to all Terraform modules when inputs = {compatibility_mode = "${local.compatibility_mode}"}
is specified).
When creating a new cluster, the locals
and remote_state
sections in the
env.hcl
file should be reviewed and updated using values appropriate to your
AWS account and VPC:
locals {
k8s_distro = "rke2-cluster"
# Specifies the type of cluster (deployTarget or collab)
cluster_type = "deployTarget"
# Required AWS Configuration
aws_region = "us-east-2"
vpc_id = "vpc-123456789abcdef01"
vpc_subnet_ids = [
"subnet-123456789abcdef01",
"subnet-123456789abcdef02",
"subnet-123456789abcdef03"
]
allowed_security_group_ids = [
"sg-123456789abcdef01"
]
# If true, this flag disables some AWS features which are not available in all AWS partitions/regions.
compatibility_mode = false
# Optionally toggle add-on modules on/off
# cluster_type = "collab" automatically toggles all of these on
# modules = {
# confluence = false
# console = false
# gitlab = false
# jira = false
# keycloak = true
# loki = true
# mattermost = true
# sonarqube = true
# velero = true
# }
# Optionally provide Add-on specific configs directly
# See each add-ons documentation for available values
# confluence_inputs = {}
# console_inputs = {}
# gitlab_inputs = {}
# jira_inputs = {}
# keycloak_inputs = {}
# loki_inputs = {}
# mattermost_inputs = {}
# sonarqube_inputs = {}
# velero_inputs = {}
# Cluster specific configuration
# See eks module documentation for available values
cluster_inputs = {
cluster_name = "env-collab"
cp_ami = "ami-123456789abcdef01"
# If you want to provide a role, define cluster_cp_iam_role and cluster_agent_iam_role for control plane
# and agent node roles, respectively.
# If left blank, Structsure will create roles to be used.
cluster_cp_iam_role = "your-role-here"
cluster_agent_iam_role = "your-role-here"
}
vpc_id = "vpc-123456789abcdef01"
vpc_subnet_ids = [
"subnet-123456789abcdef01",
"subnet-123456789abcdef02",
"subnet-123456789abcdef03"
]
...
}
remote_state {
config = {
bucket = "structsure-terraform-state"
dynamodb_table = "structsure-terraform-lock"
accesslogging_bucket_name = "structsure-logging"
}
...
}
inputs = {
aws_region = "${local.aws_region}"
compatibility_mode = "${local.compatibility_mode}"
vpc_id = "${local.vpc_id}"
}
For more information on recommended values for the env.hcl
, particularly
values suitable for a high-availability production cluster, please refer to
Production-Grade Deployment Requirements documentation.
Source Terraform Providers and Modules
This section is only applicable for offline deployments. In connected deployments, Terraform is able to download the providers and modules from the Internet.
Offline Terraform bundles for the AWS IaC repository are generated for each
release and stored in the project's package registry. Download the appropriate bundle for your version of the AWS IaC
repository, then extract the bundle to the top-level folder of your cloned AWS
IaC repository using a command like tar xzvf BUNDLE_PATH
.
The extracted bundle should create a hidden directory (viewable with ls -a
) in
the cloned repo called '.terragrunt-cache'. This directory should in turn
contain sub-folders called 'modules' and 'providers'. The downloaded modules
should automatically be used by Terragrunt if that folder exists (see the
link_offline_bundle
hook in infra-iac/common.hcl
for implementation details),
but Terragrunt will need to be directed to use the provider cache by setting the
TF_PLUGIN_CACHE_DIR
environment variable to the absolute path of the
.terragrunt-cache/providers
folder.
Configure Terragrunt S3 Backend
Terragrunt will typically be configured to use an S3 backend to store its state
information. Generally, this will be configured in the env.hcl
file for
your environment using a block, such as the following:
remote_state {
backend = "s3"
config = {
bucket = "BUCKET_NAME"
dynamodb_table = "DYNAMODB_TABLE"
key = "${basename(get_terragrunt_dir())}.tfstate"
}
}
Each Terragrunt module will source this remote_state
configuration from the
env.hcl
file so that they share a common configuration, but because of the
dynamically-generated key
name, if multiple Terragrunt modules are run, they
will each have independent states.
Terragrunt is able to configure the backend S3 bucket and DynamoDB table for you
automatically if they do not yet exist. As such, you may see the following
message when you run terragrunt init
or terragrunt run-all init
:
Remote state S3 bucket BUCKET_NAME does not exist or you don't have permissions
to access it. Would you like Terragrunt to create it? (y/n)
This may be expected if this is the first time that Terragrunt has been run with
the configured backend settings; if this is the case, select yes
to have
Terragrunt automatically create and configure the S3 bucket and DynamoDB table
for the backend. However, if this is not the first time running Terragrunt
using a particular backend configuration, most likely you do not have
permissions to access the configured S3 bucket. Try running an aws s3 ls
s3://BUCKET_NAME
from your terminal window and compare its results to the AWS
Management Console. If the bucket exists within the AWS Management Console,
then verify your instance role permissions and/or that your temporary
credentials have not expired.
Execute the Terragrunt
Depending upon whether your cluster is destined to become a deploy target or a collaboration environment, the procedure for executing the Terragrunt is somewhat different; the collaboration environment add-ons require some additional infrastructure (RDS databases, S3 buckets, etc.), which are contained in individual Terragrunt modules so that they can be independently deployed in an a-la-carte fashion. For a deploy target, the only IaC component that needs to be deployed is the RKE2 cluster itself; the following example will show the steps required to build the infrastructure for a deploy target:
cd REPO_PATH
export TF_PLUGIN_CACHE_DIR="$(pwd)/.terragrunt-cache/providers/"
export TERRAGRUNT_ENV_FILE="$(pwd)/infra-iac/envs/your-custom-env.hcl"
export WORKSPACE_NAME="your-workspace-name"
cd infra-iac/rke2-cluster
terragrunt init
terragrunt workspace select -or-create $WORKSPACE_NAME
terragrunt plan
terragrunt apply
Note the following:
- As mentioned above, the
TF_PLUGIN_CACHE_DIR
must be configured with the absolute path of the Terraform provider bundle if running in a disconnected environment. This is not required in a connected environment, but it may speed up the init process if initializing multiple modules (as for a collaboration environment), due to the shared cache. - If using a custom Terragrunt configuration file, the absolute path should be
specified using the
TERRAGRUNT_ENV_FILE
environment variable. If not specified, Terragrunt will look for a file called 'env-default.hcl' in the parent directories. - The workspace name is arbitrary, although in development, it will generally correlate to a branch name. Multiple unique clusters can be provisioned using the same Terraform state bucket by using distinct workspaces for each; if doing this, make sure that the workspace allows you to easily identify each cluster.
- It is not strictly required to plan your changes before running
terragrunt apply
, but you should generally do so in order to preview the changes to be made before proceeding. This is especially important when upgrading to modifying already-deployed infrastructure. - These steps align with the commands run automatically during the AWS IaC CI Pipelines, so they may serve as a useful reference.
Outputs
Terragrunt will store several useful values as outputs; these can be viewed
using the terragrunt output
command. For example, to configure your local
terminal to use the cluster's kubeconfig file, you may run the following:
cd REPO_PATH/infra-iac/rke2-cluster
KUBECONFIG_URL=$(terragrunt output -raw kubeconfig_url)
aws s3 cp $KUBECONFIG_URL ~/.kube/config
This will allow you to run kubectl
commands in the context of the cluster.
If multiple Terragrunt modules have been deployed, as for a collaboration environment, the outputs for all of them can be combined using a command, such as the following:
cd REPO_PATH/infra
terragrunt run-all output -json | jq -n 'reduce inputs as $i ({}; . * $i)' >
outputs.json
Additionally, Terragrunt will also create files (by default in the
infra-iac/outputs
directory) matching several of the Terraform outputs. These
will be consumed during the deployment of the Structsure package.