Customizing Alertmanager Configuration

The Alertmanager feature allows you to handle alerts sent by all ARTESCA applications. It handles the deduplication and the grouping of alerts to route them to the correct receiver.

After enabling the Alert Notification feature, you can customize the Alertmanager configuration. The following documentation only presents basic configuration usages.

For more details or advanced configuration, refer to the official Alertmanager documentation.

Prerequisites

• You must have Alert Notification configured.

• You must have a good understanding of the Alertmanager feature.

Procedure

To override the default Alertmanager configuration and implement customization, you need to edit its Cluster and Service ConfigMap metalk8s-alertmanager-config in the namespace metalk8s-monitoring under the key data.config\.yaml.

1. Open the configuration file.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
       edit configmap -n metalk8s-monitoring \
       metalk8s-alertmanager-config
   

2. Add the wanted configuration.

   In this example, we are adding the alert inhibition rules. This configuration inhibits alerts with a warning severity when the same alert with a critical severity is firing, where <alertname> is the name of the alert to inhibit.

   Important

   Do not customize the Alertmanager configuration without a good understanding of this feature and its impact. If you wish to implement a custom configuration, contact Scality support.

   apiVersion: v1
   kind: ConfigMap
   data:
     config.yaml: |-
       apiVersion: addons.metalk8s.scality.com
       kind: AlertmanagerConfig
       spec:
         notification:
           config:
             inhibit_rules:
               - source_match:
                   severity: critical
                 target_match:
                   severity: warning
                 equal:
                   - <alertname>
   

3. Export the MetalK8s version you are using as an environment variable:

   [artesca-os@bootstrap ~]$
   

   export MK8SVERSION=`sudo kubectl \
       --kubeconfig=/etc/kubernetes/admin.conf \
       get cm -n kube-system artesca-version \
       -o jsonpath="{.data.metalk8s}"`
   

4. Apply the configuration.

   Important

   Any changes made to metalk8s-alertmanager-config ConfigMap must be applied using Salt.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl exec --kubeconfig /etc/kubernetes/admin.conf \
       $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -l app=salt-master -o name) \
       -n kube-system -c salt-master -- \
       salt-run state.sls metalk8s.addons.prometheus-operator.deployed \
       saltenv=metalk8s-$MK8SVERSION
   

5. Check the configuration by testing receivers.

   This check is done by sending a mock alert. This alert remains for 5 minutes before it is automatically resolved by Alertmanager.

   [artesca-os@bootstrap ~]$
   

   curl -H 'Content-Type: application/json' \
    -d '[{"labels":{"alertname":"Mock_Alert", "severity":"critical"}}]' \
    http://$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
        get services -n metalk8s-monitoring prometheus-operator-alertmanager \
        -o jsonpath='{.spec.clusterIP}'):9093/api/v2/alerts