Prometheus logo
DevOps,  Kubernetes,  Monitoring Tools,  Prometheus

Prometheus-operator: How to add custom scrape targets

Prometheus-operator comes with pre-configured scrape targets to keep an eye on kubernetes cluster standard components. At some point, you might want to add some custom targets to monitor your application. This page shows you how to achieve it.

  • Create a yaml manifest that includes all the extra custom scrape targets you want to add:
    • Sample file: prometheus-additional.yaml
- job_name: "your_custom_job_name"
  - targets: ["your_endpoint_providing_metrics:your_port"]
  metrics_path: "/a/b/c/metrics/application"

Target configuration settings

Value of “targets” can only be a hostname or ip address (typically: Your application pod’s Service name, e.g. podname.namespace.svc.cluster.local) and the corresponding port.

By default, in case you do NOT specify the “metrics_path”, prometheus will contact http://hostname:port/metrics

In case your application provides metrics to a different path, you must provide it as value of “metrics_path”.

  • Create a Secret yaml manifest with name = additional-scrape-configs reading the content from file prometheus-additional.yaml created on step 1 above:
# kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -o yaml > additional-scrape-configs.yaml
  • Create a Secret using the yaml manifest generated on step 2 above and make sure to assign it to the same namespace in use by prometheus:
# kubectl apply -f additional-scrape-configs.yaml -n monitoring
  • Edit your Prometheus CRD (Custom Resource Definition) and add a reference to your additional scrape configs (new block: spec.additionalScrapeConfigs):
# kubectl edit prometheus/prometheus-kube-prometheus-prometheus -n monitoring

kind: Prometheus
  annotations: prometheus monitoring
  creationTimestamp: "2022-09-15T07:20:00Z"
  generation: 2
    app: kube-prometheus-stack-prometheus prometheus Helm kube-prometheus-stack 40.0.0
    chart: kube-prometheus-stack-40.0.0
    heritage: Helm
    release: prometheus
  name: prometheus-kube-prometheus-prometheus
  namespace: monitoring
  resourceVersion: "11481588"
  uid: 465362f4-a309-4022-94fb-62f5e22f4828
    key: prometheus-additional.yaml
    name: additional-scrape-configs
. . .
  • Restart kube-prometheus and kube-operator pods:
# kubectl delete -n monitoring $(kubectl get pods -o=name -n monitoring -l app=kube-prometheus-stack-operator)
# kubectl delete -n monitoring $(kubectl get pods -o=name -n monitoring -l

As soon as the new pods come up, metrics collected from your new targets will be accessible from Prometheus/Grafana.