DevOps,  DevSecOps,  Kubernetes,  Monitoring Tools,  Prometheus

How to expose kubernetes api-server metrics

Kubernetes api-server provides very interesting metrics which could make a difference when it comes to detecting potential security threats.

Accessing api-server requires a Token and a certificate. Both must be related to a ServiceAccount with sufficient permissions to access metrics endpoint. This post describes how to achieve such setup.


Before to start, make sure your current context is using “default” namespace
kubectl config set-context --current --namespace=default

Step 1: Create a new ServiceAccount

kubectl create serviceaccount metrics-explorer

Step 2: Create a new ClusterRole with sufficient permissions to access api-server metrics endpoint via HTTP GET

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
  name: metrics-explorer
- nonResourceURLs:
  - /metrics
  - /metrics/cadvisor
  - get

Step 3: Create new ClusterRoleBinding to bind the ServiceAccount with ClusterRole

kubectl create clusterrolebinding metrics-explorer:metrics-explorer --clusterrole metrics-explorer --serviceaccount default:metrics-explorer

Step 4: Export ServiceAccount’s token Secret’s name

SECRET=$(kubectl get serviceaccount ${SERVICE_ACCOUNT} -o json | jq -Mr '.secrets[].name | select(contains("token"))')

Step 5: Extract Bearer token from Secret and decode it

TOKEN=$(kubectl get secret ${SECRET} -o json | jq -Mr '.data.token' | base64 -d)

Step 6: Extract, decode and write the ca.crt to a temporary location

kubectl get secret ${SECRET} -o json | jq -Mr '.data["ca.crt"]' | base64 -d > /tmp/ca.crt

Final step: Test access to metrics endpoint

curl -s <API-SERVER>/metrics  --header "Authorization: Bearer $TOKEN" --cacert /tmp/ca.crt | less

Configuring as additional scrape target on Prometheus

Transfer the certificate file from api-server’s VM to Prometheus’ VM. (e.g. destination filename: /opt/api-server-files/ca.crt)

Save the TOKEN obtained on steps above to a file on Prometheus’ VM. (e.g. destination filename: /opt/api-server-files/api-server-token)

Edit Prometheus main configuration file (e.g. /etc/prometheus/prometheus.yml) and add the following scrape target:

  - bearer_token_file: /opt/api-server-files/api-server-token
    job_name: kubernetes-apiservers
    - targets: ['<API-SERVER-IP>:6443']
    metrics_path: '/metrics'
    scheme: https
      ca_file: /opt/api-server-files/ca.crt