Skip to main content
Version: 5.17.1

Configure Monitoring Log Storage

Structsure's IAC Terragrunt <env>.hcl file is default deployed with the cluster_inputs variable persistent set to false; if persistent is set to true, monitoring applications will be created with persistent storage PVCs.

locals {
  cluster_inputs = {
    persistent = true
  ...
  }
}

The size of your cluster, and/or the amount of logs you generate, may necessitate configuration of monitoring storage values; see how-to-override-bigbang-values for more information.

This document details how to configure, and override, the monitoring applications default values.

Alertmanager​

monitoring:
  values:
    alertmanager:
      alertmanagerSpec:
        storage:
          volumeClaimTemplate:
            spec:
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 20Gi

Grafana​

tip

Grafana's default strategy is set to Recreate so the EBS can be freed for the new pod; this will cause some downtime during upgrades and restarts. You can choose to use a ReadWriteMany Storage Class (such as EFS) and set the strategy to rollout to scale and avoid downtime.

grafana:
  values:
    deploymentStrategy:
      type: Recreate
    persistence:
      type: pvc
      enabled: true
      # storageClassName: default
      accessModes:
        - ReadWriteOnce
      finalizers:
        - kubernetes.io/pvc-protection
      size: 20Gi

Prometheus​

caution

Non-POSIX compliant filesystems are not supported for Prometheus' local storage as unrecoverable corruptions may occur. NFS filesystems (including AWS's EFS) are not supported. NFS could be POSIX-compliant, but most implementations are not. It is strongly recommended to use a local filesystem for reliability.

Retention time, defaults to 15d. Supported units: y, w, d, h, m, s, ms.

monitoring:
  values:
    prometheus:
      prometheusSpec:
        persistentVolumeClaimRetentionPolicy:
          whenDeleted: Retain
          whenScaled: Retain
        storageSpec:
          retentionSize: "64GB"
          retention: 10d
          volumeClaimTemplate:
            spec:
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 75Gi

Big Bang Monitoring Overview