Skip to main content
Version: 5.21.0

RKE2 Config Reference

Structsure Cluster IAC

Overview

This project contains the IaC to deploy an RKE2 Based K8s cluster.

Requirements

NameVersion
terraform>= 0.14
archive~> 2.4.0
aws<= 5.22.0
local~> 2.4.0
null~> 3.2.2
postgresql~> 1.21.0
random~> 3.6.0
sops~> 1.0.0
template~> 2.2.0
time~> 0.9.0
tls~> 4.0.5

Providers

NameVersion
aws5.22.0
local2.4.1
null3.2.2
random3.6.1
tls4.0.5

Modules

NameSourceVersion
app_clb../modules/loadbalancern/a
app_nlbterraform-aws-modules/alb/aws~> 9.0.0
log_bucket_protected../modules/structsure_s3_protectedn/a
log_bucket_unprotected../modules/structsure_s3n/a
rke2_agents../modules/rke2-aws-tf/modules/agent-nodepooln/a
rke2_cp../modules/rke2-aws-tfn/a
sso_clb../modules/loadbalancern/a
sso_nlbterraform-aws-modules/alb/aws~> 9.0.0
userdata_s3../modules/structsure_s3n/a
zarf_registry_s3terraform-aws-modules/s3-bucket/aws~> 3.0

Resources

NameType
aws_autoscaling_attachment.app-httpresource
aws_autoscaling_attachment.app-httpsresource
aws_autoscaling_attachment.sso-httpresource
aws_autoscaling_attachment.sso-httpsresource
aws_iam_policy.allow_ebsresource
aws_iam_policy.allow_kmsresource
aws_iam_policy.allow_s3_etcd_snapshotsresource
aws_iam_policy.allow_s3_userdataresource
aws_iam_policy.allow_s3_zarfresource
aws_iam_role_policy_attachment.allow_s3_etcd_snapshotsresource
aws_iam_role_policy_attachment.cp_allow_ebsresource
aws_iam_role_policy_attachment.cp_allow_kmsresource
aws_iam_role_policy_attachment.cp_allow_s3_userdataresource
aws_iam_role_policy_attachment.cp_allow_s3_zarfresource
aws_iam_role_policy_attachment.cp_allow_ssm_cloudwatchresource
aws_iam_role_policy_attachment.cp_allow_ssm_coreresource
aws_iam_role_policy_attachment.cp_collab_allow_efsresource
aws_iam_role_policy_attachment.va_allow_ebsresource
aws_iam_role_policy_attachment.va_allow_kmsresource
aws_iam_role_policy_attachment.va_allow_s3_userdataresource
aws_iam_role_policy_attachment.va_allow_s3_zarfresource
aws_iam_role_policy_attachment.va_allow_ssm_cloudwatchresource
aws_iam_role_policy_attachment.va_allow_ssm_coreresource
aws_iam_role_policy_attachment.va_collab_allow_efsresource
aws_key_pair.ssh_public_keyresource
aws_s3_bucket_logging.rke2-s3-loggingresource
aws_s3_bucket_versioning.rke2-s3-versioningresource
aws_s3_object.userdata_filesresource
aws_security_group_rule.loadbalancer_ingressresource
aws_security_group_rule.rke2_agent_ssh_ruleresource
aws_security_group_rule.rke2_cp_ssh_ruleresource
local_file.bigbang_valuesresource
local_file.zarf_ca_bundleresource
local_file.zarf_init_configresource
null_resource.agent_instance_name_length_checkerresource
null_resource.cp_instance_name_length_checkerresource
random_password.zarf_registry_pull_passwordresource
random_shuffle.cp_lb_subnetsresource
random_shuffle.va_lb_subnetsresource
random_shuffle.va_nlb_subnetsresource
tls_private_key.sshresource
aws_ami.rke2data source
aws_iam_policy_document.allow_ebsdata source
aws_iam_policy_document.allow_kmsdata source
aws_iam_policy_document.allow_s3_etcd_snapshotsdata source
aws_iam_policy_document.allow_s3_userdatadata source
aws_iam_policy_document.allow_s3_zarfdata source
aws_partition.currentdata source
aws_region.currentdata source
aws_s3_object.kubeconfigdata source
aws_service.s3data source
aws_vpc.vpcdata source

Inputs

NameDescriptionTypeDefaultRequired
additional_imagesA list of S3 URIs. Enables downloading of image artifacts to nodeslist(string)[]no
allowed_ingress_cidr_blockAllows ingress from this cidr block.string""no
ami_filtersList of AMI filters used to select the AMIlist(object({ name = string values = list(string) }))[]no
ami_idAMI ID to be used for both control plane and agent nodes, unless overriddenstring""no
ami_most_recentSelect the most recent version of the AMIbooltrueno
ami_ownersList of AWS account IDs used in AMI lookup filterlist(string)[ "241559654725", "171179903432" ]no
application_nlb_enableToggles the creation of the Application Network Load Balancer off/onbooltrueno
aws_regionAWS Region used to configure the AWS providerstringn/ayes
bigbang_values_filenameFilename of the locally-created big bang values includestring"bigbang-values-rke2.yaml"no
blackhole_githubConfigure CoreDNS to blackhole api.github.comboolfalseno
bucket_force_destroyBool that allows S3 buckets to be destroyed when not emptyboolfalseno
clamav_notification_email_addressName of the email address to receive clamav detectionsstringn/ayes
cloud_configObject that enables and determines the contents of the cloud-config fileobject({ enabled = string value = string }){ "enabled": false, "value": "" }no
cluster_agent_iam_roleIAM role for the agents to usestring"InstanceOpsRole"no
cluster_cp_iam_roleIAM role for the Control Plane to usestring"InstanceOpsRole"no
cluster_extra_tagsAdditional tags to add to all cluster nodesmap(string){}no
common_userdata_variablesExtra environmental variables that need to be passed between userdata scripts.list(object({ name = string value = string }))[]no
compatibility_modeIf enabled, this flag disables some AWS features which are not available in all AWS partitions/regions.booltrueno
config_output_dirPath to directory where local config files should be outputstring"."no
cp_allow_sshToggles SSH security group rule for structsure Control Plane nodes on or offbooltrueno
cp_allowed_cidrsServer pool security group allowed cidr rangeslist(string)[ "0.0.0.0/0" ]no
cp_amiControl plane AMI ID, overrides the ami_id variablestringnullno
cp_bdm_delete_on_termControl Plane block device mapping delete on termination settingbooltrueno
cp_bdm_device_nameName of root block device mappingstring"/dev/xvda"no
cp_bdm_encryptedControl Plane block device mapping encryption settingbooltrueno
cp_bdm_sizeControl Plane block device mapping sizenumber40no
cp_bdm_typeControl Plane block device mapping typestring"gp2"no
cp_cz_elbToggle between controlplane cross zone load balancingbooltrueno
cp_downloadToggle best effort download of rke2 dependencies (rke2 and aws cli), if disabled, dependencies are assumed to exist in $PATHboolfalseno
cp_ebs_kms_key_idAWS KMS key ID for ebs related operations.stringn/ayes
cp_extra_block_device_mappingList of objects to configure addional diskslist(map(string))[]no
cp_extra_security_group_idsList of additional security group IDs to attach to control plane nodeslist(string)[]no
cp_extra_tagsAdditional tags to add to control plane nodesmap(string){}no
cp_instance_typeServer pool instance typestring"t3a.large"no
cp_internal_elbToggle between public or private control plane load balancerbooltrueno
cp_lb_haNeed control plane load balancers to have high availability (multi-az)booltrueno
cp_rke2_configServer pool additional configuration passed as rke2 config file, see here for full list of optionsstring"disable:\n - \"rke2-ingress-nginx\"\nkubelet-arg:\n - \"cloud-provider=external\"\n - \"protect-kernel-defaults=true\"\n - \"streaming-connection-idle-timeout=5m\"\n - \"tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"\nkube-apiserver-arg:\n - \"enable-admission-plugins=NodeRestriction\"\n - \"request-timeout=60s\"\n - \"audit-log-mode=blocking-strict\"\n - \"tls-min-version=VersionTLS12\"\n - \"tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"\n - \"anonymous-auth=false\"\n - \"authorization-mode=RBAC,Node\"\nkube-scheduler-arg:\n - \"tls-min-version=VersionTLS12\"\n - \"tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"\n - \"bind-address=0.0.0.0\"\nkube-controller-manager-arg:\n - \"cloud-provider=external\"\n - \"configure-cloud-routes=false\"\n - \"tls-min-version=VersionTLS12\"\n - \"tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"\n - \"bind-address=0.0.0.0\"\nkube-proxy-arg:\n - \"metrics-bind-address=0.0.0.0\"\npod-security-admission-config-file: /etc/rancher/rke2/rke2-pss-custom.yaml\nprofile: cis-1.23\nnode-taint:\n # work around https://github.com/rancher/rke2/issues/508n - \"node-role.kubernetes.io/master:PreferNoSchedule\"\nwrite-kubeconfig-mode: \"0640\"\nsecrets-encryption: true\nselinux: true\netcd-expose-metrics: true\ndisable-cloud-controller: true\n"no
cp_serversNumber of servers to createnumbern/ayes
cp_spotToggle spot requests for server poolboolfalseno
cp_ssh_cidr_blocksCIDR block allowed ssh access to structsure Control Plane nodeslist(string)[ "10.0.0.0/8" ]no
cp_subnetsList of subnet IDs to create resources inlist(string)n/ayes
cp_userdata_variablesExtra environmental variables specific to control plane that need to be passed between userdata scripts.list(object({ name = string value = string }))[]no
cp_wait_for_capacity_timeoutHow long Terraform should wait for ASG instances to be healthy before timing out.string"20m"no
create_app_elbBool that determines if a classic elb is createdboolfalseno
create_sso_elbBool that determines if a classic elb is createdboolfalseno
domain_nameName of the domain.string"dsop.structsure.io"no
enable_ccmToggle enabling the cluster as aws aware, this will ensure the appropriate IAM policies are presentboolfalseno
etcd_backupAutomatically back up etcd snapshots to the rke2 token bucketbooltrueno
helm_chartsList of objects describing helm charts to be deployedlist(map(any))[]no
log_bucket_prevent_destroyBool that allows the S3 log bucket to be destroyedboolfalseno
mail_relay_from_addressName of the from email address required to use the ac2sp relaystringn/ayes
name_prefixName to prefix resources with.string"collab"no
name_prefix_include_workspaceToogle to include workspace in name prefixboolfalseno
notification_alert_emailName of the email address.stringn/ayes
persistentFlag to set the deployment to persistent or ephemeralboolfalseno
registries_configRKE2 registries.yaml configurationany{}no
resize_disksEnables automatic resizing of disks in userdata scriptbooltrueno
root_casList containing root certificate authorities (optionally base64 encoded)list( object( { name = string cert = string } ) )[]no
sso_passthrough_enableToggles SSO (Keycloak) to use the passthrough Network Load Balancerboolfalseno
userdata_commandA string containing shell commands which will be executed during the provisioning process for nodes. These commands will run on both control plane and agent nodes, and will execute at the end of the pre_userdata section of the provisioning, prior to the starting of the RKE2 service.string""no
userdata_command_filesList of relative paths (relative to the userdata_file_dir) to shell (sh) scripts which should be executed as part of the instance provisioning process (after being downloaded from the S3 userdata bucket). These scripts will run on both control plane and agent nodes, and will execute at the end of the pre_userdata section of the provisioning, prior to the starting of the RKE2 service.list(string)[]no
userdata_files_dirThe path to a directory containing files to upload to the userdata files S3 bucket. If these conflict with files in the "${path.module}/files" directory, the files in the userdata_files_dir will overwrite the defaults in the "${path.module}/files" bucket.string"fakeemptydir1234"no
userdata_files_s3_prefixThis prefix will be added to the object keys for all userdata files uploaded to S3.string""no
va_allow_sshToggles SSH security group rule for structsure Agent nodes on or offboolfalseno
va_amiAgent node AMI ID, overrides the ami_id variablestringnullno
va_asgNode pool AutoScalingGroup scaling definitionobject({ min = number max = number desired = number })n/ayes
va_bdm_delete_on_termstructsure Agents block device mapping delete on termination settingbooltrueno
va_bdm_device_nameName of root block device mappingstring"/dev/xvda"no
va_bdm_encryptedstructsure Agents block device mapping encryption settingbooltrueno
va_bdm_sizestructsure Agents block device mapping sizenumber80no
va_bdm_typestructsure Agents block device mapping typestring"gp2"no
va_downloadToggle best effort download of rke2 dependencies (rke2 and aws cli), if disabled, dependencies are assumed to exist in $PATHboolfalseno
va_ebs_kms_key_idAWS KMS key ID for ebs related operations.stringn/ayes
va_enable_autoscalerToggle configure the nodepool for cluster autoscaler, this will ensure the appropriate IAM policies are present, you are still responsible for ensuring cluster autoscaler is installedbooltrueno
va_extra_block_device_mappingList of objects to configure addional diskslist(map(string))[]no
va_extra_security_group_idsList of additional security group IDs to attach to structsure agent nodeslist(string)[]no
va_extra_tagsAdditional tags to add to structsure agent nodesmap(string){}no
va_instance_typeServer pool instance typestring"t3a.large"no
va_lb_haNeed application and sso load balancers to have high availability (multi-az)booltrueno
va_nameNodepool namestring"cbc2_structsure-agents"no
va_nlb_enable_deletion_protectionPrevent the NLB(s) from being deletedbooltrueno
va_nlb_preserve_client_ipToggles the preserve_client_ip setting on/off. Will always be false if compatibility_mode is setbooltrueno
va_nlb_stickiness_configApplication NLB Stickiness settings. Will always be an empty set if compatibility_mode is setany{ "duration": 3600, "enabled": true, "type": "source_ip" }no
va_rke2_configAgent pool additional configuration passed as rke2 config file, see here for full list of optionsstring"kubelet-arg:\n - \"cloud-provider=external\"\n - \"protect-kernel-defaults=true\"\n - \"streaming-connection-idle-timeout=5m\"\n - \"tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"\nnode-label:\n - \"genericAgent=true\"\nkube-proxy-arg:\n - \"metrics-bind-address=0.0.0.0\"\nselinux: true\n"no
va_spotToggle spot requests for node poolboolfalseno
va_ssh_cidr_blocksCIDR block allowed ssh access to structsure Agent nodeslist(string)[ "10.0.0.0/8" ]no
va_subnetsList of subnet IDs to create resources inlist(string)n/ayes
va_userdata_variablesExtra environmental variables specific to agents that need to be passed between userdata scripts.list(object({ name = string value = string }))[]no
va_wait_for_capacity_timeoutHow long Terraform should wait for ASG instances to be healthy before timing out.string"30m"no
vault_passthrough_enableToggles Vault to use the passthrough Network Load Balancerboolfalseno
vpc_idVPC ID to create resources instringn/ayes
zarf_init_ca_bundle_filenameFilename of the locally-created zarf init config yaml filestring"zarf-ca-bundle.pem"no
zarf_init_config_filenameFilename of the locally-created zarf init config yaml filestring"zarf-init-config.yaml"no
zarf_registry_enabledFlag to enable the creation of zarf registry bucket and configurationbooltrueno
zarf_registry_ironbank_mirror_enabledtoggle for configuring a containerd mirror to point registry1.dso.mil to zarf's registrybooltrueno
zarf_registry_nodeportzarf registry node port; must be between 30000-32767number31999no
zarf_registry_pull_passwordzarf registry pull password; if not supplied, a random one will be generatedstring""no
zarf_registry_pull_usernamezarf registry pull usernamestring"zarf-pull"no
zarf_registry_redirect_disableDisable the registry redirectstring"true"no
zarf_registry_shared_bucket_idName of an existing shared zarf registry bucketstring"shared-zarf-registry"no
zarf_registry_shared_enabledFlag to enable using an existing shared zarf registry bucketboolfalseno

Outputs

NameDescription
app_elb_dns_nameDNS name for the application ELB
app_nlb_dns_nameDNS name for the app NLB
aws_dns_suffixOutput the DNS suffix of the current aws partition
aws_partitionOutput the name of the current aws partition
aws_regionAWS region used
bucket_prefixPrefix used to create gitlab buckets
cluster_nameName of the created cluster
cluster_security_groupSecurity group shared by cluster nodes, this is different than nodepool security groups
create_agent_roleBool that determines if an IAM role is created
desired_node_countTotal desired node count across all node groups
domainDNS name of the cluster that was created
kubeconfigThe content of the kubeconfig file
kubeconfig_dataMap of the connection information contained within the kubeconfig
kubeconfig_urlS3 URL to the rke2.yaml kubeconfig
log_bucket_nameS3 log bucket name for storing S3 server logs
minimum_node_countTotal minimum node count across all node groups
node_group_namesn/a
rke2_agents_iam_roleIAM role of rke2_agents node pool
rke2_bucketS3 bucket for rke2 storage
ssh_private_keyThe SSH private key configured as an authorized key on nodes.
sso_elb_dns_nameDNS name for the SSO ELB
sso_nlb_dns_nameDNS name for the sso NLB
target_group_arnsn/a
zarf_registry_bucket_idName of the bucket created for use with the Zarf registry