Prometheus-operator: How to configure email notifications for alerts

This guide relates to Prometheus-operator.

Whenever an alerting rule on Prometheus starts firing, the issue is only visible either accessing Prometheus web UI or from Grafana. In case you want it to trigger email notifications as well, follow this guide.

• Create a new Secret to store your SMTP server’s authentication password (only if it requires authentication)
  • Sample yaml manifest:

apiVersion: v1
data:
  password: abcde==
kind: Secret
metadata:
  name: prometheus-smtp-settings
  namespace: monitoring
type: Opaque

• Create a new AlertmanagerConfig object
  • Sample yaml manifest (replace SMTP settings according to your SMTP server):

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: prometheus-alertmanager-email-configs
  namespace: monitoring
  labels:
    alertmanagerConfig: email
spec:
  route:
    groupBy: ['alertname']
    groupWait: 10s
    groupInterval: 10s
    repeatInterval: 5m
    receiver: 'email'
  receivers:
  - name: 'email'
    emailConfigs:
    - to: 'test@test.com'
      from: 'test@test.com'
      smarthost: smtp.test.com:587
      authUsername: test@test.com
      authPassword:
        name: prometheus-smtp-settings
        key: password
      requireTLS: true

Filtering alerts based on their label

In case you want to filter alerts that should be routed to the receiver (“email”, from sample above), you can add a filtering rule as child of spec.route

Sample:

. . .
spec:
  route:
    groupBy: ['alertname']
    groupWait: 10s
    groupInterval: 10s
    repeatInterval: 5m
    receiver: 'email'
    matchers:
      - severity=~"critical|warning"
. . .

• Restart prometheus alertmanager:

$ kubectl delete -n monitoring $(kubectl get pods -n monitoring -l alertmanager=prometheus-kube-prometheus-alertmanager -o=name)