How to Setup a Highly Available Zarf Registry

In order to support updating/rotating Kubernetes nodes, the Zarf registry must be highly available. Out-of-the-box, the Zarf registry is configured to create a persistent volume claim (PVC) using the default storage class. If this storage class is not ReadWriteMany (RWX), or cannot be mounted by multiple agent nodes at the same time, you cannot easily move the registry between nodes without downtime. The suggested solution in Amazon Web Services (AWS) environments is to utilize Simple Storage Service (S3) instead of this PVC.

S3 Storage Driver

The registry container image has support for an S3 storage driver. This is the best solution for a cluster deployed to AWS. Utilizing S3 removes the need for an RWX or replicated block storage class and allows for multiple registry instances to read/write images simultaneously.

Prerequisites

Zarf Version

Support for configuring the registry with extra environment variables and no PVC was added in Zarf v0.29.1. Support for providing a Certificate Authority (CA) bundle via extra configmaps, volumes, and volumeMounts is currently pending approval and release in PR 2008. If you need to deploy the registry using this method, you can build your own init package by executing the following commands:

git clone -b registry-ca https://github.com/AbrohamLincoln/zarf.git
cd zarf
zarf package create . --set AGENT_IMAGE_TAG=v0.29.1 --confirm

This will result in a file named zarf-init-amd64-v0.29.1.tar.zst that can be used to perform the deployment with a custom CA bundle, as depicted in this document.

S3 Bucket

This configuration requires an S3 bucket. Create a bucket that adheres to your organizational standards, and retain the bucket name and region for configuration purposes.

Identity and Access Management (IAM) Policy

The registry requires IAM permissions in order to use the S3 bucket. The following policy is the minimum set of permissions required for pushing and pulling images. Please note that S3_BUCKET_NAME placeholder must be updated with your bucket name. Also note that the aws partition in the Resource field may need to be updated for your environment.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME/*"
    }
  ]
}

IAM Credentials

The registry needs credentials to interact with the S3 Application Programming Interface (API). There are three main options: IAM Instance Profiles, IAM User Access/Secret Keys, and IAM Roles for Service Accounts (IRSA).

While all three are valid options, the most common solution to date is to use the Identity and Access Management (IAM) Instance Profile. Currently, the Registry2 container does not support Instance Metadata Service Version 2 (IMDSv2), which can be an issue when using Instance Profiles as your credentials. If you choose this option, ensure the IMDSv1 is enabled by issuing the following AWS command-line interface (CLI) command, and update it with your instance's ID.

a2aws ec2 modify-instance-metadata-options \
    --instance-id i-0123456789abcdef0 \
    --http-tokens optional \
    --http-endpoint enabled

CA Bundle

If you are deploying a registry in an AWS partition that does not use a commercial CA, you will need to create a custom trusted root bundle and supply it to the registry.

You can use OpenSSL to print out the certificate chain presented by the S3 API. In order to do this, you will need the URL for your bucket. In this example, we will use: https://my-example-bucket.s3.us-east-2.amazonaws.com. If you are running these commands, ensure to substitute in your bucket and S3 endpoint.

echo "Q" | openssl s_client -connect my-example-bucket.s3.us-east-2.amazonaws.com:443 -showcerts

The output should look similar to the following:

Collapsed output

% echo "Q" | openssl s_client -connect my-example-bucket.s3.us-east-2.amazonaws.com:443 -showcerts
CONNECTED(00000006)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01
verify return:1
depth=0 CN = *.s3.us-east-2.amazonaws.com
verify return:1
---
Certificate chain
 0 s:CN = *.s3.us-east-2.amazonaws.com
   i:C = US, O = Amazon, CN = Amazon RSA 2048 M01
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 11 00:00:00 2023 GMT; NotAfter: Feb 28 23:59:59 2024 GMT
-----BEGIN CERTIFICATE-----
MIIH+DCCBuCgAwIBAgIQAV//CoXG02wxH6TzRI212TANBgkqhkiG9w0BAQsFADA8
MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRwwGgYDVQQDExNBbWF6b24g
UlNBIDIwNDggTTAxMB4XDTIzMDQxMTAwMDAwMFoXDTI0MDIyODIzNTk1OVowJzEl
MCMGA1UEAwwcKi5zMy51cy1lYXN0LTIuYW1hem9uYXdzLmNvbTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBALtKvpIoha2Vi3xmw+UlGKSztNLaq7lKHPhM
RDnCZ+JpYQ7kCNIPTyRBfE9qdMtMBBvxypHziWcBftwZSg54tRzvxcjtkqYGtCJG
baW1bw2MTiRlzdYD/sIxy5D8Ox6GOlRk7CSmqgJGTEkgaKD36X1DVW8vMow6bPvR
mWCAVcV2I3h02u+1stMcjasUnh9zPtXEvJ2jKFtafG4kjXndMr6Jrrqf/MFwiIlB
Cyf7oWNjx2XBko3qe026KXoIy6xRwD/XVj0fOAFP/Is4nHyH/5QAOs8P2FyUUZkH
aazZbWmHw4khedlFKZc5WY/5rYw3X3MwgjHGsQzpU24yVj+l7ssCAwEAAaOCBQkw
ggUFMB8GA1UdIwQYMBaAFIG4DmOKiRIY5fo7O1CVn+blkBOFMB0GA1UdDgQWBBQ0
Tro5JLzZ4Ujzm90wp5HJZzJTgTCCAjkGA1UdEQSCAjAwggIsghwqLnMzLnVzLWVh
c3QtMi5hbWF6b25hd3MuY29tghpzMy51cy1lYXN0LTIuYW1hem9uYXdzLmNvbYIc
Ki5zMy11cy1lYXN0LTIuYW1hem9uYXdzLmNvbYIaczMtdXMtZWFzdC0yLmFtYXpv
bmF3cy5jb22CJiouczMuZHVhbHN0YWNrLnVzLWVhc3QtMi5hbWF6b25hd3MuY29t
giRzMy5kdWFsc3RhY2sudXMtZWFzdC0yLmFtYXpvbmF3cy5jb22CEiouczMuYW1h
em9uYXdzLmNvbYIkKi5zMy1jb250cm9sLnVzLWVhc3QtMi5hbWF6b25hd3MuY29t
giJzMy1jb250cm9sLnVzLWVhc3QtMi5hbWF6b25hd3MuY29tgi4qLnMzLWNvbnRy
b2wuZHVhbHN0YWNrLnVzLWVhc3QtMi5hbWF6b25hd3MuY29tgixzMy1jb250cm9s
LmR1YWxzdGFjay51cy1lYXN0LTIuYW1hem9uYXdzLmNvbYIoKi5zMy1hY2Nlc3Nw
b2ludC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbYIyKi5zMy1hY2Nlc3Nwb2ludC5k
dWFsc3RhY2sudXMtZWFzdC0yLmFtYXpvbmF3cy5jb22CJyouczMtZGVwcmVjYXRl
ZC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbYIlczMtZGVwcmVjYXRlZC51cy1lYXN0
LTIuYW1hem9uYXdzLmNvbTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMDsGA1UdHwQ0MDIwMKAuoCyGKmh0dHA6Ly9jcmwucjJt
MDEuYW1hem9udHJ1c3QuY29tL3IybTAxLmNybDATBgNVHSAEDDAKMAgGBmeBDAEC
ATB1BggrBgEFBQcBAQRpMGcwLQYIKwYBBQUHMAGGIWh0dHA6Ly9vY3NwLnIybTAx
LmFtYXpvbnRydXN0LmNvbTA2BggrBgEFBQcwAoYqaHR0cDovL2NydC5yMm0wMS5h
bWF6b250cnVzdC5jb20vcjJtMDEuY2VyMAwGA1UdEwEB/wQCMAAwggF+BgorBgEE
AdZ5AgQCBIIBbgSCAWoBaAB3AHb/iD8KtvuVUcJhzPWHujS0pM27KdxoQgqf5mdM
Wjp0AAABh3HcOAUAAAQDAEgwRgIhAPExm6sLpTPMYqd36BgjWDKTPG99jcnzlWt2
7oak5trqAiEAq5NmE3cp46JsLN0KcRg5PoSQQ/33XfDx1LUuL8vIiIsAdQBIsONr
2qZHNA/lagL6nTDrHFIBy1bdLIHZu7+rOdiEcwAAAYdx3DfdAAAEAwBGMEQCIDvX
lUMIqInGg03gf73OYv6JIR2TFm6h3UZal2ZM83ApAiBMLQrnbBua+Kvnhw3oDODn
rKBr8aFOwjhtzaptjSydKQB2ANq2v2s/tbYin5vCu1xr6HCRcWy7UYSFNL2kPTBI
1/urAAABh3HcN5IAAAQDAEcwRQIgFO2b5QnAFLrr7FElhfaqvu2zaFmEghd/1+q2
hszzToYCIQDlisKe7oyF+IsN7FvsNWnpz5FQeIci2kpSJNR5/WzGNDANBgkqhkiG
9w0BAQsFAAOCAQEAcE+lAnVubUyKiim9DBQuL6HQD6RXmtssGHhG40faOoCVoViS
mnqCYsK7nNpNl/zafBWgbpoxk6z6Mqykaa/uwdqsYyhcuGJyb9Gj0SvPbp2S29Hf
GKL4IKqzdQ9LK8fuhXJyKZkFmap7MTm3ax40n3uS5BVv7yvpe7lEFVKmiq0ezWA+
+ITxftu2Y+Tk50M3jugnTm1bmcSi7NXN3ZzZRXkES3KvytWV6bM22+cp1Dphson0
NOsz8mnJ1yz258ReyA2D1hdXA8p5JzaRXA/67wvHpCpfVMt3UIUfuy+fsb4o8zQE
nyUP4VRplbDDXNXiy/xJaCkdftn2AikQNSJA7g==
-----END CERTIFICATE-----
 1 s:C = US, O = Amazon, CN = Amazon RSA 2048 M01
   i:C = US, O = Amazon, CN = Amazon Root CA 1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
-----BEGIN CERTIFICATE-----
MIIEXjCCA0agAwIBAgITB3MSOAudZoijOx7Zv5zNpo4ODzANBgkqhkiG9w0BAQsF
ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6
b24gUm9vdCBDQSAxMB4XDTIyMDgyMzIyMjEyOFoXDTMwMDgyMzIyMjEyOFowPDEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEcMBoGA1UEAxMTQW1hem9uIFJT
QSAyMDQ4IE0wMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOtxLKnL
H4gokjIwr4pXD3i3NyWVVYesZ1yX0yLI2qIUZ2t88Gfa4gMqs1YSXca1R/lnCKeT
epWSGA+0+fkQNpp/L4C2T7oTTsddUx7g3ZYzByDTlrwS5HRQQqEFE3O1T5tEJP4t
f+28IoXsNiEzl3UGzicYgtzj2cWCB41eJgEmJmcf2T8TzzK6a614ZPyq/w4CPAff
nAV4coz96nW3AyiE2uhuB4zQUIXvgVSycW7sbWLvj5TDXunEpNCRwC4kkZjK7rol
jtT2cbb7W2s4Bkg3R42G3PLqBvt2N32e/0JOTViCk8/iccJ4sXqrS1uUN4iB5Nmv
JK74csVl+0u0UecCAwEAAaOCAVowggFWMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYD
VR0PAQH/BAQDAgGGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNV
HQ4EFgQUgbgOY4qJEhjl+js7UJWf5uWQE4UwHwYDVR0jBBgwFoAUhBjMhTTsvAyU
lC4IWZzHshBOCggwewYIKwYBBQUHAQEEbzBtMC8GCCsGAQUFBzABhiNodHRwOi8v
b2NzcC5yb290Y2ExLmFtYXpvbnRydXN0LmNvbTA6BggrBgEFBQcwAoYuaHR0cDov
L2NydC5yb290Y2ExLmFtYXpvbnRydXN0LmNvbS9yb290Y2ExLmNlcjA/BgNVHR8E
ODA2MDSgMqAwhi5odHRwOi8vY3JsLnJvb3RjYTEuYW1hem9udHJ1c3QuY29tL3Jv
b3RjYTEuY3JsMBMGA1UdIAQMMAowCAYGZ4EMAQIBMA0GCSqGSIb3DQEBCwUAA4IB
AQCtAN4CBSMuBjJitGuxlBbkEUDeK/pZwTXv4KqPK0G50fOHOQAd8j21p0cMBgbG
kfMHVwLU7b0XwZCav0h1ogdPMN1KakK1DT0VwA/+hFvGPJnMV1Kx2G4S1ZaSk0uU
5QfoiYIIano01J5k4T2HapKQmmOhS/iPtuo00wW+IMLeBuKMn3OLn005hcrOGTad
hcmeyfhQP7Z+iKHvyoQGi1C0ClymHETx/chhQGDyYSWqB/THwnN15AwLQo0E5V9E
SJlbe4mBlqeInUsNYugExNf+tOiybcrswBy8OFsd34XOW3rjSUtsuafd9AWySa3h
xRRrwszrzX/WWGm6wyB+f7C4
-----END CERTIFICATE-----
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgITBn+USionzfP6wq4rAfkI7rnExjANBgkqhkiG9w0BAQsF
ADCBmDELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNj
b3R0c2RhbGUxJTAjBgNVBAoTHFN0YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4x
OzA5BgNVBAMTMlN0YXJmaWVsZCBTZXJ2aWNlcyBSb290IENlcnRpZmljYXRlIEF1
dGhvcml0eSAtIEcyMB4XDTE1MDUyNTEyMDAwMFoXDTM3MTIzMTAxMDAwMFowOTEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv
b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj
ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM
9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw
IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6
VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L
93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm
jgSubJrIqg0CAwEAAaOCATEwggEtMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/
BAQDAgGGMB0GA1UdDgQWBBSEGMyFNOy8DJSULghZnMeyEE4KCDAfBgNVHSMEGDAW
gBScXwDfqgHXMCs4iKK4bUqc8hGRgzB4BggrBgEFBQcBAQRsMGowLgYIKwYBBQUH
MAGGImh0dHA6Ly9vY3NwLnJvb3RnMi5hbWF6b250cnVzdC5jb20wOAYIKwYBBQUH
MAKGLGh0dHA6Ly9jcnQucm9vdGcyLmFtYXpvbnRydXN0LmNvbS9yb290ZzIuY2Vy
MD0GA1UdHwQ2MDQwMqAwoC6GLGh0dHA6Ly9jcmwucm9vdGcyLmFtYXpvbnRydXN0
LmNvbS9yb290ZzIuY3JsMBEGA1UdIAQKMAgwBgYEVR0gADANBgkqhkiG9w0BAQsF
AAOCAQEAYjdCXLwQtT6LLOkMm2xF4gcAevnFWAu5CIw+7bMlPLVvUOTNNWqnkzSW
MiGpSESrnO09tKpzbeR/FoCJbM8oAxiDR3mjEH4wW6w7sGDgd9QIpuEdfF7Au/ma
eyKdpwAJfqxGF4PcnCZXmTA5YpaP7dreqsXMGz7KQ2hsVxa81Q4gLv7/wmpdLqBK
bRRYh5TmOTFffHPLkIhqhBGWJ6bt2YFGpn6jcgAKUj6DiAdjd4lpFw85hdKrCEVN
0FE6/V1dN2RMfjCyVSRCnTawXZwXgWHxyvkQAiSr6w10kY17RSlQOYiypok1JR4U
akcjMS9cmvqtmg5iUaQqqcT5NJ0hGA==
-----END CERTIFICATE-----
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT
-----BEGIN CERTIFICATE-----
MIIEdTCCA12gAwIBAgIJAKcOSkw0grd/MA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
BAYTAlVTMSUwIwYDVQQKExxTdGFyZmllbGQgVGVjaG5vbG9naWVzLCBJbmMuMTIw
MAYDVQQLEylTdGFyZmllbGQgQ2xhc3MgMiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTAeFw0wOTA5MDIwMDAwMDBaFw0zNDA2MjgxNzM5MTZaMIGYMQswCQYDVQQGEwJV
UzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTElMCMGA1UE
ChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjE7MDkGA1UEAxMyU3RhcmZp
ZWxkIFNlcnZpY2VzIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IC0gRzIwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVDDrEKvlO4vW+GZdfjohTsR8/
y8+fIBNtKTrID30892t2OGPZNmCom15cAICyL1l/9of5JUOG52kbUpqQ4XHj2C0N
Tm/2yEnZtvMaVq4rtnQU68/7JuMauh2WLmo7WJSJR1b/JaCTcFOD2oR0FMNnngRo
Ot+OQFodSk7PQ5E751bWAHDLUu57fa4657wx+UX2wmDPE1kCK4DMNEffud6QZW0C
zyyRpqbn3oUYSXxmTqM6bam17jQuug0DuDPfR+uxa40l2ZvOgdFFRjKWcIfeAg5J
Q4W2bHO7ZOphQazJ1FTfhy/HIrImzJ9ZVGif/L4qL8RVHHVAYBeFAlU5i38FAgMB
AAGjgfAwge0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0O
BBYEFJxfAN+qAdcwKziIorhtSpzyEZGDMB8GA1UdIwQYMBaAFL9ft9HO3R+G9FtV
rNzXEMIOqYjnME8GCCsGAQUFBwEBBEMwQTAcBggrBgEFBQcwAYYQaHR0cDovL28u
c3MyLnVzLzAhBggrBgEFBQcwAoYVaHR0cDovL3guc3MyLnVzL3guY2VyMCYGA1Ud
HwQfMB0wG6AZoBeGFWh0dHA6Ly9zLnNzMi51cy9yLmNybDARBgNVHSAECjAIMAYG
BFUdIAAwDQYJKoZIhvcNAQELBQADggEBACMd44pXyn3pF3lM8R5V/cxTbj5HD9/G
VfKyBDbtgB9TxF00KGu+x1X8Z+rLP3+QsjPNG1gQggL4+C/1E2DUBc7xgQjB3ad1
l08YuW3e95ORCLp+QCztweq7dp4zBncdDQh/U90bZKuCJ/Fp1U1ervShw3WnWEQt
8jxwmKy6abaVd38PMV4s/KCHOkdp8Hlf9BRUpJVeEXgSYCfOn8J3/yNTd126/+pZ
59vPr5KW7ySaNRB6nJHGDn2Z9j8Z3/VyVOEVqQdZe4O/Ui5GjLIAZHYcSNPYeehu
VsyuLAOQ1xk4meTKCRlb/weWsKh/NEnfVqn3sF/tM+2MR7cwA130A4w=
-----END CERTIFICATE-----
---
Server certificate
subject=CN = *.s3.us-east-2.amazonaws.com
issuer=C = US, O = Amazon, CN = Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 6007 bytes and written 472 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: BAEB242C7400579F3056EDC377DD7F1E8EA5659DE3083D89EB122442FD3A45E1
    Session-ID-ctx:
    Master-Key: 47757FD478D2C4B232D139CA2E179520581B7AABB6C40907314CF1AEF6EBD46ABF52E5ABD65E185074CF201675AF4E37
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1693933781
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
DONE

Use every certificate presented by the endpoint, except the one with a subject that matches the endpoint URL. In our example, the following subjects are in the certificate chain:

• 0 s:CN = *.s3.us-east-2.amazonaws.com
• 1 s:C = US, O = Amazon, CN = Amazon RSA 2048 M01
• 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
• 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2

In this case, certificate 0 matches our subject. All other certificates will be included in our CA bundle. You can simply concatenate the certificates together, preferably prefixing them with a comment, so you know which is which. Continuing with our example, our CA bundle would end up looking similar to the following:

Collapsed CA Bundle Example

# Amazon RSA 2048 M01
-----BEGIN CERTIFICATE-----
MIIEXjCCA0agAwIBAgITB3MSOAudZoijOx7Zv5zNpo4ODzANBgkqhkiG9w0BAQsF
ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6
b24gUm9vdCBDQSAxMB4XDTIyMDgyMzIyMjEyOFoXDTMwMDgyMzIyMjEyOFowPDEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEcMBoGA1UEAxMTQW1hem9uIFJT
QSAyMDQ4IE0wMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOtxLKnL
H4gokjIwr4pXD3i3NyWVVYesZ1yX0yLI2qIUZ2t88Gfa4gMqs1YSXca1R/lnCKeT
epWSGA+0+fkQNpp/L4C2T7oTTsddUx7g3ZYzByDTlrwS5HRQQqEFE3O1T5tEJP4t
f+28IoXsNiEzl3UGzicYgtzj2cWCB41eJgEmJmcf2T8TzzK6a614ZPyq/w4CPAff
nAV4coz96nW3AyiE2uhuB4zQUIXvgVSycW7sbWLvj5TDXunEpNCRwC4kkZjK7rol
jtT2cbb7W2s4Bkg3R42G3PLqBvt2N32e/0JOTViCk8/iccJ4sXqrS1uUN4iB5Nmv
JK74csVl+0u0UecCAwEAAaOCAVowggFWMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYD
VR0PAQH/BAQDAgGGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNV
HQ4EFgQUgbgOY4qJEhjl+js7UJWf5uWQE4UwHwYDVR0jBBgwFoAUhBjMhTTsvAyU
lC4IWZzHshBOCggwewYIKwYBBQUHAQEEbzBtMC8GCCsGAQUFBzABhiNodHRwOi8v
b2NzcC5yb290Y2ExLmFtYXpvbnRydXN0LmNvbTA6BggrBgEFBQcwAoYuaHR0cDov
L2NydC5yb290Y2ExLmFtYXpvbnRydXN0LmNvbS9yb290Y2ExLmNlcjA/BgNVHR8E
ODA2MDSgMqAwhi5odHRwOi8vY3JsLnJvb3RjYTEuYW1hem9udHJ1c3QuY29tL3Jv
b3RjYTEuY3JsMBMGA1UdIAQMMAowCAYGZ4EMAQIBMA0GCSqGSIb3DQEBCwUAA4IB
AQCtAN4CBSMuBjJitGuxlBbkEUDeK/pZwTXv4KqPK0G50fOHOQAd8j21p0cMBgbG
kfMHVwLU7b0XwZCav0h1ogdPMN1KakK1DT0VwA/+hFvGPJnMV1Kx2G4S1ZaSk0uU
5QfoiYIIano01J5k4T2HapKQmmOhS/iPtuo00wW+IMLeBuKMn3OLn005hcrOGTad
hcmeyfhQP7Z+iKHvyoQGi1C0ClymHETx/chhQGDyYSWqB/THwnN15AwLQo0E5V9E
SJlbe4mBlqeInUsNYugExNf+tOiybcrswBy8OFsd34XOW3rjSUtsuafd9AWySa3h
xRRrwszrzX/WWGm6wyB+f7C4
-----END CERTIFICATE-----
# Amazon Root CA 1
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgITBn+USionzfP6wq4rAfkI7rnExjANBgkqhkiG9w0BAQsF
ADCBmDELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNj
b3R0c2RhbGUxJTAjBgNVBAoTHFN0YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4x
OzA5BgNVBAMTMlN0YXJmaWVsZCBTZXJ2aWNlcyBSb290IENlcnRpZmljYXRlIEF1
dGhvcml0eSAtIEcyMB4XDTE1MDUyNTEyMDAwMFoXDTM3MTIzMTAxMDAwMFowOTEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv
b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj
ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM
9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw
IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6
VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L
93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm
jgSubJrIqg0CAwEAAaOCATEwggEtMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/
BAQDAgGGMB0GA1UdDgQWBBSEGMyFNOy8DJSULghZnMeyEE4KCDAfBgNVHSMEGDAW
gBScXwDfqgHXMCs4iKK4bUqc8hGRgzB4BggrBgEFBQcBAQRsMGowLgYIKwYBBQUH
MAGGImh0dHA6Ly9vY3NwLnJvb3RnMi5hbWF6b250cnVzdC5jb20wOAYIKwYBBQUH
MAKGLGh0dHA6Ly9jcnQucm9vdGcyLmFtYXpvbnRydXN0LmNvbS9yb290ZzIuY2Vy
MD0GA1UdHwQ2MDQwMqAwoC6GLGh0dHA6Ly9jcmwucm9vdGcyLmFtYXpvbnRydXN0
LmNvbS9yb290ZzIuY3JsMBEGA1UdIAQKMAgwBgYEVR0gADANBgkqhkiG9w0BAQsF
AAOCAQEAYjdCXLwQtT6LLOkMm2xF4gcAevnFWAu5CIw+7bMlPLVvUOTNNWqnkzSW
MiGpSESrnO09tKpzbeR/FoCJbM8oAxiDR3mjEH4wW6w7sGDgd9QIpuEdfF7Au/ma
eyKdpwAJfqxGF4PcnCZXmTA5YpaP7dreqsXMGz7KQ2hsVxa81Q4gLv7/wmpdLqBK
bRRYh5TmOTFffHPLkIhqhBGWJ6bt2YFGpn6jcgAKUj6DiAdjd4lpFw85hdKrCEVN
0FE6/V1dN2RMfjCyVSRCnTawXZwXgWHxyvkQAiSr6w10kY17RSlQOYiypok1JR4U
akcjMS9cmvqtmg5iUaQqqcT5NJ0hGA==
-----END CERTIFICATE-----
# Starfield Services Root Certificate Authority - G2
-----BEGIN CERTIFICATE-----
MIIEdTCCA12gAwIBAgIJAKcOSkw0grd/MA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
BAYTAlVTMSUwIwYDVQQKExxTdGFyZmllbGQgVGVjaG5vbG9naWVzLCBJbmMuMTIw
MAYDVQQLEylTdGFyZmllbGQgQ2xhc3MgMiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTAeFw0wOTA5MDIwMDAwMDBaFw0zNDA2MjgxNzM5MTZaMIGYMQswCQYDVQQGEwJV
UzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTElMCMGA1UE
ChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjE7MDkGA1UEAxMyU3RhcmZp
ZWxkIFNlcnZpY2VzIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IC0gRzIwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVDDrEKvlO4vW+GZdfjohTsR8/
y8+fIBNtKTrID30892t2OGPZNmCom15cAICyL1l/9of5JUOG52kbUpqQ4XHj2C0N
Tm/2yEnZtvMaVq4rtnQU68/7JuMauh2WLmo7WJSJR1b/JaCTcFOD2oR0FMNnngRo
Ot+OQFodSk7PQ5E751bWAHDLUu57fa4657wx+UX2wmDPE1kCK4DMNEffud6QZW0C
zyyRpqbn3oUYSXxmTqM6bam17jQuug0DuDPfR+uxa40l2ZvOgdFFRjKWcIfeAg5J
Q4W2bHO7ZOphQazJ1FTfhy/HIrImzJ9ZVGif/L4qL8RVHHVAYBeFAlU5i38FAgMB
AAGjgfAwge0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0O
BBYEFJxfAN+qAdcwKziIorhtSpzyEZGDMB8GA1UdIwQYMBaAFL9ft9HO3R+G9FtV
rNzXEMIOqYjnME8GCCsGAQUFBwEBBEMwQTAcBggrBgEFBQcwAYYQaHR0cDovL28u
c3MyLnVzLzAhBggrBgEFBQcwAoYVaHR0cDovL3guc3MyLnVzL3guY2VyMCYGA1Ud
HwQfMB0wG6AZoBeGFWh0dHA6Ly9zLnNzMi51cy9yLmNybDARBgNVHSAECjAIMAYG
BFUdIAAwDQYJKoZIhvcNAQELBQADggEBACMd44pXyn3pF3lM8R5V/cxTbj5HD9/G
VfKyBDbtgB9TxF00KGu+x1X8Z+rLP3+QsjPNG1gQggL4+C/1E2DUBc7xgQjB3ad1
l08YuW3e95ORCLp+QCztweq7dp4zBncdDQh/U90bZKuCJ/Fp1U1ervShw3WnWEQt
8jxwmKy6abaVd38PMV4s/KCHOkdp8Hlf9BRUpJVeEXgSYCfOn8J3/yNTd126/+pZ
59vPr5KW7ySaNRB6nJHGDn2Z9j8Z3/VyVOEVqQdZe4O/Ui5GjLIAZHYcSNPYeehu
VsyuLAOQ1xk4meTKCRlb/weWsKh/NEnfVqn3sF/tM+2MR7cwA130A4w=
-----END CERTIFICATE-----

Configuration

The default Zarf registry configuration can be overridden by either supplying values at execution via the --set argument or creating a configuration file. The configuration file is recommended in this scenario, especially if you need to supply a CA bundle. The following configuration file can be used as an example for how to configure the registry to be backed by S3. Note that supplying the REGISTRY_CA_BUNDLE variable is not necessary in commercial, AWS, or GovCloud.

package:
  deploy:
    set:
      REGISTRY_PVC_ENABLED: false
      REGISTRY_EXTRA_ENVS: |
        - name: REGISTRY_STORAGE
          value: s3
        - name: REGISTRY_STORAGE_S3_BUCKET
          value: my-registry-bucket
        - name: REGISTRY_STORAGE_S3_REGION
          value: us-east-2
      REGISTRY_CA_BUNDLE: |
        # Root CA 1
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
        # Intermediate CA 1
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
        # Intermediate CA 2
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----

You can use an IAM Access and Secret Key, as well:

package:
  deploy:
    set:
      REGISTRY_PVC_ENABLED: false
      REGISTRY_EXTRA_ENVS: |
        - name: REGISTRY_STORAGE
          value: s3
        - name: REGISTRY_STORAGE_S3_ACCESSKEY
          value: AKIAIOSFODNN7EXAMPLE
        - name: REGISTRY_STORAGE_S3_SECRETKEY
          value: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

If you need to supply a custom S3 API endpoint, run the following:

package:
  deploy:
    set:
      REGISTRY_PVC_ENABLED: false
      REGISTRY_EXTRA_ENVS: |
        - name: REGISTRY_STORAGE
          value: s3
        - name: REGISTRY_STORAGE_S3_REGIONENDPOINT
          value: https://s3.us-east-2.amazonaws.com

Additional S3 Driver parameters can be found here. All of these parameters can be specified as an environment variable by prefixing them with REGISTRY_STORAGE_S3_ and capitalizing the parameter name. For example, the storageclass parameter can be provided as the REGISTRY_STORAGE_S3_STORAGECLASS environment variable.