Prometheus-operator: How to modify or delete pre-installed alerts

This guide relates to Prometheus-operator.

Prometheus-operator operator comes with a set of pre-installed alerts. This page shows you how to edit/remove them.

• Prometheus operator stores rules into PrometheusRule objects. In case you found a rule by looking into the Prometheus web UI, first thing you have to do is determine the group name:

• Now, retrieve the list of all PrometheusRule objects available within the cluster:

root@odin:~/prometheus# kubectl get prometheusrule -n monitoring
NAME                                                              AGE
prometheus-icas-rules                                             7d2h
prometheus-kube-prometheus-alertmanager.rules                     22d
prometheus-kube-prometheus-config-reloaders                       22d
prometheus-kube-prometheus-etcd                                   22d
prometheus-kube-prometheus-general.rules                          22d
prometheus-kube-prometheus-k8s.rules                              22d
prometheus-kube-prometheus-kube-apiserver-availability.rules      22d
prometheus-kube-prometheus-kube-apiserver-burnrate.rules          22d
prometheus-kube-prometheus-kube-apiserver-histogram.rules         22d
prometheus-kube-prometheus-kube-apiserver-slos                    22d
prometheus-kube-prometheus-kube-prometheus-general.rules          22d
prometheus-kube-prometheus-kube-prometheus-node-recording.rules   22d
prometheus-kube-prometheus-kube-scheduler.rules                   22d
prometheus-kube-prometheus-kube-state-metrics                     22d
prometheus-kube-prometheus-kubelet.rules                          22d
prometheus-kube-prometheus-kubernetes-apps                        22d
prometheus-kube-prometheus-kubernetes-resources                   22d
prometheus-kube-prometheus-kubernetes-storage                     22d
prometheus-kube-prometheus-kubernetes-system                      22d
prometheus-kube-prometheus-kubernetes-system-apiserver            22d
prometheus-kube-prometheus-kubernetes-system-controller-manager   22d
prometheus-kube-prometheus-kubernetes-system-kube-proxy           22d
prometheus-kube-prometheus-kubernetes-system-kubelet              22d
prometheus-kube-prometheus-kubernetes-system-scheduler            22d
prometheus-kube-prometheus-node-exporter                          22d    <------
prometheus-kube-prometheus-node-exporter.rules                    22d
prometheus-kube-prometheus-node-network                           22d
prometheus-kube-prometheus-node.rules                             22d
prometheus-kube-prometheus-prometheus                             22d
prometheus-kube-prometheus-prometheus-operator                    22d

• Now you can edit the object and change/delete the rule:

root@odin:~/prometheus# kubectl edit prometheusrule/prometheus-kube-prometheus-node-exporter -n monitoring
 
 
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2022-09-15T07:20:01Z"
  generation: 1
  labels:
    app: kube-prometheus-stack
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 40.0.0
    chart: kube-prometheus-stack-40.0.0
    heritage: Helm
    release: prometheus
  name: prometheus-kube-prometheus-node-exporter
  namespace: monitoring
  resourceVersion: "8740458"
  uid: c0a48da3-f7dd-4677-8ed5-2339e5d8d8c1
spec:
  groups:
  - name: node-exporter
    rules:
    - alert: NodeFilesystemSpaceFillingUp
      annotations:
        description: Filesystem on {{ $labels.device }} at {{ $labels.instance }}
          has only {{ printf "%.2f" $value }}% available space left and is filling
          up.
        runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemspacefillingup
        summary: Filesystem is predicted to run out of space within the next 24 hours.
      expr: |-
        (
          node_filesystem_avail_bytes{job="node-exporter",fstype!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!=""} * 100 < 15
        and
          predict_linear(node_filesystem_avail_bytes{job="node-exporter",fstype!=""}[6h], 24*60*60) < 0
        and
          node_filesystem_readonly{job="node-exporter",fstype!=""} == 0
        )
      for: 1h
      labels:
        severity: warning
. . .