Configuration Instructions

The following are step-by-step instructions on how to configure Structsure.

Keycloak

Before setting up applications, Keycloak needs to be set up first.

Creating the Realm, Administrator (Admin) Groups, and Clients

Enable Fine-Grained Permissions

Under client details, in the Permissions tab, toggle Permissions enabled.

Create the Realm

Upon initial install, you will receive a default admin username and password. Use these to log into Keycloak.

1. Click on master in the drop-down menu on top of the left pane.
2. Click on the Create Realm button.
3. Enter structsure for Realm name and click the Create button.
4. In the upper right corner, click on the drop-down menu for admin and click Manage account.
5. Select Signing in under Account security and update the password for the admin account.

Create Structsure Admin Group

1. In the structsure realm, create a top-level group named _structsureAdmins under the Groups option on the left column menu.
2. Inside that group, on the Role mapping tab, click Assign role.
3. Filter by clients, search for realm-admin, toggle the check box, and click the Assign button.

Creating Clients in Realms

The following are general instructions for creating a client and can be applied in jira, confluence, authservice, logging, gitlab, grafana, mattermost, sonarqube, argocd, and monitoring. Create a unique client name within each of these applications. Note: The client name may need to be updated later, if instructed in a specific application section.

1. While you are in the structsure realm, click on Clients under Manage in the left pane.
2. Click Create client.
3. Enter client name for Client ID.
4. Click on the Save button.
5. The Settings page for the new client will display.
6. Enter * for Valid Redirect URIs and toggle on Client authentication.
7. Click on the Save button.

Creating the Console Client

1. In the structsure realm, click on Clients under Manage in the left pane.
2. Click Create client.
3. Ensure the Client type is openid-connect.
4. Enter a unique client name (e.g., structsure-console) for Client ID.
5. Leave Client authentication toggled off, and ensure Direct access grants is unchecked.
6. Leave the Root URL blank and set Valid Redirect URIs to https://console.${DOMAIN}.
7. Click on the Save button.

To configure this client:

1. In the Settings tab, ensure Standard flow is toggled.
2. In the OpenID Connect Compatibility Modes section, ensure Use Refresh Tokens is ON.
3. Click Save at the bottom of the screen.

In the Mappers Tab:

1. If structsure-tools-mapper is already an option in the list, no further action is needed.
2. Otherwise, click Create at the top right.
3. Enter structsure-tools-mapper in the Name field.
4. Select User Attribute for Mapper Type.
5. Enter tools for User Attribute and Token Claim Name.
6. Set Claim JSON Type to String.
7. Ensure Add to ID token is OFF.
8. Ensure Add to access token and Add to userinfo is ON.
9. Ensure Multivalued and Aggregate attribute values is ON.
10. Click Save.

In the Client Scopes Tab:

Ensure Full Scope Allowed is ON.

Creating Client Scopes

Gitlab needs to be added in the Client Scopes.

note

The client scope has to be spelled exactly as Gitlab as this name is what GitLab is configured to request.

1. While you are in the structsure realm, click on Client Scopes.
2. Enter Gitlab for Name, and click on the Save button.
3. Go to the Mappers tab.
4. Click on Add Builtin
5. Search for username, and click the check box for username in search results.
6. Click on the Add selected button.
7. Repeat for profile, full name, and email built-in mappers to add, as well.
8. Go to Clients, and click on the same client name received for GitLab.
9. Go to Client Scopes.
10. Click on Gitlab under Available Client Scopes, and click Add selected to add it to Assigned Default Client Scopes.

Optional: Configuring Keycloak Common Access Card (CAC)/Smart Card Authentication

Background Information

Keycloak supports logging in with an X.509 client certificate if you have configured the server to use mutual SSL authentication. As a result, Keycloak must be configured to use passthrough ingress.

Additionally, Keycloak must be provided with Certificate Authorities (CAs) to trust when verifying client certificates.

Within Structsure, Keycloak can easily be configured to support CAC authentication. Some additional settings must be applied to the realm to enable users to login with their CAC.

Users will only be prompted for their CAC if they have it inserted before visiting the login page. If it is not present when visiting the login page, the user will be prompted for username and password or an error if username and password authentication has been disabled.

Configuring Keycloak for Passthrough Ingress

Enabling the necessary IaC components and resulting bigbang configuration is as simple as toggling this variable in the IaC inputs:

locals {
  cluster_inputs = {
    sso_passthrough_enable = true
  }
}

Enabling CAC Login

These steps are largely the same as Keycloak's documentation with some minor changes, given the format of CAC certificates.

1. While you are in the structsure realm, click on Authentication under Configure in the left pane.
2. Select the browser flow.
3. In the top right, select Action -> Duplicate.
   1. Name: Browser CAC Auth
   2. Description: Browser based authentication with CAC authentication
4. Click Duplicate.
5. Click Add step.
6. Search and select X509/Validate Username Form.
7. Click Add.
8. Drag X509/Validate Username Form block to be the second block just under Cookie.
9. In X509/Validate Username Form's Requirement drop-down, select Alternative.
10. Click on X509/Validate Username Form's settings (gear icon) and update the following:
    1. Alias: CAC Auth
    2. A regular expression to extract user identity: CN=(?:.*?)(\d*)(?:,|$)
    3. A name of user attribute: DOD_ID
    4. Check certificate validity: toggle on.
11. Click Save.
12. Click Action -> Bind flow:
    1. Choose binding type: Browser flow
13. Click Save.

Associating Users with CACs

Now that Keycloak can prompt users for their CAC, Keycloak needs to know what user is associated to which CAC.

For each user you want to associate:

1. While you are in the structsure realm, click on Users under Manage in the left pane.
2. Select the desired user.
3. Click the Attributes tab.
4. Add an attribute:
   1. Key: DOD_ID
   2. Value: The DOD ID number found on the back of their CAC.
5. Click Save.

Troubleshooting

Users Are Not Being Prompted for Their CAC

There are a few likely causes for this:

1. The Browser flow authentication flow, created in the steps above, is not bound for the structsure realm.
   1. Please ensure the authentication flow is bound so that users will be prompted for their CAC.
2. User has visited the login page before inserting their CAC.
   1. Have the user restart their browser or try visiting the login page from an incognito window.
3. Custom Keycloak configuration has overwritten settings necessary for CAC authentication.
   1. Please review the default configuration provided by Structsure and Keycloak documentation to determine what was lost.
4. If you are providing a custom Truststore for validating smart cards in addition to CACs:
   1. Please ensure the custom Truststore also includes the DoD CAs required for CAC authentication.

X509 certificate authentication's failed. Invalid user

This error usually means the user attempted to login with their CAC, and Keycloak was unable to associate it to an account.

Please review their DoD ID and ensure it is set correctly within their user account attributes.

Jira

As Jira is used to manage users in Confluence, Jira needs to be set up before Confluence.

Instructions for setting up Jira

1. Go to the Jira website provided.
2. Follow Wizard to enter the license key.
3. Enter the data for the initial admin account.
4. Skip the configuring email.
5. Go into the admin Applications tab and install Jira Software.
6. Set up the groups in Jira for syncing to Confluence: a. confluence-users b. confluence-administrators
7. Add the Keycloak structsure-console user to the above two user groups.
8. Wait for the users to sync from Jira to Confluence.
9. Follow the connecting Confluence section to get the Jira side configured.

Confluence

Instructions for setting up Confluence

1. Go to the Confluence website provided.
2. Follow the Wizard to enter the license key.
3. Do the initial setup with Confluence-managed users and set your admin.
4. Under Users and security, select admin, and add a directory named Atlassian Jira. a. Input the server URL for Jira. b. Add read/write permissions.
5. Manager Users and Groups within Jira: a. Configure a system administrator account. b. Create a space name. Note: See Connecting Confluence to Jira applications for User Management here for instructions. c. Log in to Jira with the admin account. d. Go to Administration -> User Management -> Jira user server and click on Add application. e. Enter confluence for Application name. f. Enter a password that was generated for Password. g. Enter 0.0.0.0/0 for IP Addresses, and click Save. h. Log into Confluence with its admin account. i. Go to Administration > General Configuration > User directories. j. Add a directory and select type Atlassian Crowd. k. Enter Jira Server for Name. l. Enter the HTTP address for the server as the Server URL. m. Enter Confluence for Application Name. n. Enter the password that was defined for the Confluence application in the Jira settings. o. Enter 15 for Synchronization Interval to synchronize every 15 minutes. p. Click Test Settings, which should show the message "Connection test successful". q. Click Save and Test to finalize the configuration.
6. Log into Confluence with the structsure-console user and validate that it is an admin account.

Confluence's Collaborative Editing feature has intentionally been disabled in Structsure due to auditing concerns. As a result, the confluence-synchrony pods have been disabled and Synchrony Connectivity health checks will show as failures in the Confluence UI. Some additional steps need to be taken after intial setup to completely disable Collaborative Editing. Please do the following when logged in as a Confluence administrator:

1. Click the gear icon in top right of Confluence UI.
2. Click General configuration.
3. In the left-hand panel, click Collaborative Editing.
4. Click Change mode.
5. Select Off.
6. Click Change.

note

You may need to disable Collaborative Editing twice before it shows it is disabled in the UI.

Mattermost

Mattermost needs to be set up prior to setting up Single Sign-On (SSO).

Instructions for setting up Mattermost

1. Register and sign in as the structsure-console user.
2. Create the Organization's "Welcome". Note: Everyone will have access to this, so avoid creating an Organization that is intended for use. a. Skip the rest of the Wizard set up.
3. In the System Console, enable access tokens: a. System Console -> Integrations -> Integration Management b. In the profile, create a PAT and save it to give to Console.
4. Keep this session open so that it can bestow permission on an SSO user.

Gitlab

1. Go to the Gitlab website provided.
2. Verify the login works as expected, using the initial password that is generated in the gitlab-gitlab-initial-root-password secret.

Removing areas of concerns in Gitlab

The following includes areas of concern for the GitLab configuration.

• SSO Configuration
• Disabling Self-Registration
• Disabling Shared Gitlab-runner
• Disabling Public Projects
• Account Limitations
• Sign-up Restrictions
• External Authorization
• Repository Mirroring
• Spam/Anti-bot Protection
• Import and Export Rate limits

Configuring Gitlab via the UI

SSO Configuration

Documentation from Big Bang and GitLab.

This functionality is being generated by the helm charts for the GitLab install.

Disabling Self-Registration

Documentation from GitLab.

As an admin, select Menu -> Admin, under Settings -> General -> Sign-up restrictions. The settings for enabling/disabling Self Registration are located within.

Turn off Sign-up enabled.

Disable Shared Gitlab-runner

Documentation from GitLab.

As an admin, select Menu -> Admin, under the Settings -> CI/CD -> Continuous Integration and Deployment. The settings for Auto DevOps, shared runners, and artifacts are located within.

Turn the following off:

• Default to Auto DevOps pipeline for all projects.
• Enable shared runners for new projects.
• Enable pipeline suggestion banner.

Disable Public Project

Documentation from GitLab.

As an admin, select Menu -> Admin, under the Settings -> General -> Visibility and access controls. The settings for visibility are located within.

Set the following:

• Default project creation protection -> Maintainers
• Restricted visibility levels -> Public
• Note: If the public level is selected, it prevents outside users from seeing in and inside users from giving outside visibility.
• Enable Git Access protocols to Only HTTP(S)
• Disable Project export enabled.

Account Limitations

As an admin, select Menu -> Admin, under the Settings -> General -> Account and Limit. The settings for Account/Limits are located within.

Set the following:

• Uncheck User OAuth Applications.
• Uncheck Prompt users to upload SSH keys.
• Set Personal Access Token Prefix to 'structsure-'.
• Set Default project limits to 0.
• Check Deactivate dormant users after 90 days of inactivity.
• Uncheck the box for allow new users to create top-level groups.
• Uncheck the box for allow users to delete their own accounts.

External Authorization

Documentation from GitLab.

As an admin, select Menu -> Admin, under the Settings -> General -> External authorization. The settings for external authorizations are located within.

Repository Mirroring

Documentation from GitLab.

As an admin, select Menu -> Admin, under the Settings -> Repository -> Repository Mirroring. The setting for mirroring repositories is located within.

Set the following:

• Uncheck Allow Repository mirroring configuration.

Spam/Anti-bot Protection

As an admin, select Menu -> Admin, under the Settings -> Reporting -> Spam and Anti-bot Protection. The setting for Spam/Anti-bot protection is located within.

Set the following:

• Enable reCAPTCHA.
• Enable reCAPTCHA for login.
• Limit sign-in from multiple IP addresses.

Import and Export Rate limits

As an admin, select Menu -> Admin, under the Settings -> Network -> User and IP rate limits. The setting for Import/Export Rate limits is located within.

• Enable all Rate Limits

Vault

Instructions for setting up Vault

Configuring Vault Infrastructure as Code (IaC)

Vault IaC is disabled by default. It must be enabled and applied, as follows, before enabling and deploying Vault in the Zarf package.

Vault needs to use mutual SSL authentication. As a result, it must be configured to use passthrough ingress. Also, the Vault IaC module itself needs to be enabled.

Enabling the necessary IaC components and resulting Big Bang configuration is as simple as toggling these variables in the IaC inputs:

locals {
  cluster_inputs = {
    vault_passthrough_enable = true
  }

  modules = {
    vault = true
  }

  vault_inputs = {
    vault_kms_seal_enabled = true  # Enables AWS KMS key auto-unseal in Vault. If set to false, we
                                   # use the shamir key, which requires manual unseal.
  }
}

Initialize and Unseal Vault

When Vault is first installed in a new cluster, it will require some manual initialization, which will create some keys that need to be safely stored for later recovery. This initialization only needs to be done in one pod, vault-vault-0. Shell into this pod (choosing the vault container if given the option) and initialize Vault as shown in the following commands:

kubectl exec --stdin=true --tty=true vault-vault-0 -n vault  -- /bin/bash
vault operator init

This command could take up to a minute to complete, but it will give out five recovery keys and an Initial Root Token. Immediately copy the recovery keys and the Initial Root Token and save them in a safe space.

danger

Losing recovery keys can result in loss of data.

Then run:

vault login

This will prompt for the Initial Root Token. Paste it in (it won't show) and hit return, which should give a success message.

To check if this has initialized the other pods, run:

vault operator raft list-peers

This will ensure it returns all the vault pods.

Next run:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  issuer="https://kubernetes.default.svc.cluster.local"

This will enable Kubernetes, and they should separately give success notices.

Write a Vault policy for Prometheus with:

vault policy write prometheus-monitoring - << EOF
    path "/sys/metrics" {
      capabilities = ["read"]
    }
EOF

Then attach the policy to the existing monitoring-monitoring-kube-prometheus ServiceAccount with:

vault write auth/kubernetes/role/prometheus \
      bound_service_account_names=monitoring-monitoring-kube-prometheus \
      bound_service_account_namespaces=monitoring \
      policies="default,prometheus-monitoring" \
      ttl="15m"

Exiting this shell should show that all vault pods are up and running, as this initialization should automatically copy to the other vault pods. If there is still a Prometheus pod in the monitoring namespace that isn't running, delete it so it can come up again with the new settings.

If, for whatever reason, it is desired that Vault be reset/uninitialized, delete all of the PVCs for Vault, then delete the pods for vault-# and repeat the above instructions.

Set Up Single Sign-On (SSO)

Each of the following clients will need SSO set up in order to use Keycloak.

Confluence, Jira, Grafana, Mattermost, SonarQube, Argo CD

Confluence

1. Install the miniOrange plugin from Atlassian Marketplace.
2. Configure the miniOrange license.
3. Follow the instructions given by miniOrange. Big Bang also has its own set of instructions that overlap with miniOrange.

Jira

1. Install the miniOrange plugin from Atlassian Marketplace.
2. Configure the miniOrange license.
3. Follow the instructions given by miniOrange. Big Bang also has its own set of instructions that overlap with miniOrange.

Grafana

1. Follow the instructions given by Big Bang.

Mattermost

1. Follow the instructions given by Big Bang.
2. Turn on the SSO flag and provide client information.
3. Create a dedicated mapper in Keycloak (may not be needed anymore).
4. Do not turn it on until Console is set up.

Sonarqube

1. Follow the instructions given by Big Bang.
2. Download the metadata to get the certificate.
3. Add the Client ID and the base64 certificate into the SSO: section.

Argo CD

1. Follow the instructions given by Big Bang.
2. Add a Groups mapper to the dedicated client scopes as a Token Mapper for Group Membership (which is not the default that enables roles).
3. Create an ArgoCD scope and attach it, as well.
4. Hook that into the SSO configuration.

A handy but not final quick start would be the following code block to let the Structsure admins rule Argo CD:

sso:
  enabled: true
  provider_name: Keycloak
  groups: |
    g, _structsureAdmins, role:admin

Scale Up Horizontal Pod Autoscaling (HPA)

Jira and Confluence require that only one pod be running during their initial setup. Afterwards, Jira and Confluence need to be configured for High Availability by scaling their statefulset / HPA.

Jira and Confluence

Jira

The maximum replica count for Jira's HPA is already configured to allow up to three pods. The Big Bang values included as an output from the IaC code provide a commented out value for setting the minimum replica count. Simply uncomment the following:

packages:
  jira:
    values:
      # replicaCount: 2

• Verify the health checks by going to the top right Settings gear -> General configuration -> Left panel -> Troubleshooting and support tools.
• Verify clustering by going to Clustering.

Confluence

The maximum replica count for Confluence's HPA is already configured to allow up to three pods. The Big Bang values included as an output from the IaC code provide a commented out value for setting the minimum replica count. Simply uncomment the following:

packages:
  confluence:
    values:
      # replicaCount: 2

• Verify the Confluence pod replication on the cluster by running kubectl get all -n confluence. This should show that the horizontalpodautoscaler has a MINPODS of 2 and a MAXPODS of 3.
• Verify the health checks by going to the top right Settings gear -> General configuration -> Left panel -> Troubleshooting and support tools.
• Verify clustering by going to Clustering.

Initialize Console

To initialize Console, the following clients need to be set up to connect to Console.

Jira, Confluence, GitLab, and Mattermost

Jira/Confluence

If Confluence is set up to use Jira's user database, then Jira and Confluence share the same users. The following addresses these console environment variables:

- `JIRA_USERNAME`
- `JIRA_PERSONAL_ACCESS_TOKEN`
- `CONFLUENCE_PERSONAL_ACCESS_TOKEN`

Create a Jira/Confluence user:

1. Log in to Jira with an admin-level account.
2. Go to User Management in the admin panel.
3. Generate a strong password: openssl rand -base64 64 | tr -d '\n' | pbcopy.
4. Create user as: a. Email address: Use an email address designated for Console. b. Full name: console. c. Username: console. d. Password: Use generated password.
5. Edit user groups for console to add: a. jira-administrators b. confluence-users c. confluence-administrators

Create Personal Access Token:

1. Log in as console.
2. Click the user icon in the top right corner, and go to Profile.
3. Create Personal Access Token.
4. Save token.

Repeat the Create Personal Access Token steps for Confluence.

GitLab

The following addresses use these console environment variables:

- `GITLAB_PERSONAL_ACCESS_TOKEN`

Create GitLab user:

1. Log into GitLab with an admin-level account.
2. Go to Users in the admin panel.
3. Create user as: a. Name: console. b. Username: console. c. Email: Use an email address designated for Console. d. Access Level: Administrator.
4. Generate a strong password: openssl rand -base64 64 | tr -d '\n' | pbcopy.
5. Manually set the password on the account.

Create Personal Access Token:

1. Impersonate or log in as console.
2. Click the user icon in the top right corner, and select Preferences.
3. Click Access Tokens on the left-hand panel.
4. Create a Personal Access Token with the sudo scope.
5. Save token.

Mattermost

The following addresses use these console environment variables:

- `MATTERMOST_ACCESS_TOKEN`
- `MATTERMOST_PERSONAL_ACCESS_TOKEN`

Currently, the above variables are the same token.

Assign the PAT (saved when Mattermost was set up) to both MATTERMOST_ACCESS_TOKEN and MATTERMOST_PERSONAL_ACCESS_TOKEN.