How to expose kubernetes api-server metrics

Kubernetes api-server provides very interesting metrics which could make a difference when it comes to detecting potential security threats.

Accessing api-server requires a Token and a certificate. Both must be related to a ServiceAccount with sufficient permissions to access metrics endpoint. This post describes how to achieve such setup.

Namespace

Before to start, make sure your current context is using “default” namespace

kubectl config set-context --current --namespace=default

Step 1: Create a new ServiceAccount

kubectl create serviceaccount metrics-explorer

Step 2: Create a new ClusterRole with sufficient permissions to access api-server metrics endpoint via HTTP GET

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: metrics-explorer
rules:
- nonResourceURLs:
  - /metrics
  - /metrics/cadvisor
  verbs:
  - get

Step 3: Create new ClusterRoleBinding to bind the ServiceAccount with ClusterRole

kubectl create clusterrolebinding metrics-explorer:metrics-explorer --clusterrole metrics-explorer --serviceaccount default:metrics-explorer

Step 4: Export ServiceAccount’s token Secret’s name

SERVICE_ACCOUNT=metrics-explorer
SECRET=$(kubectl get serviceaccount ${SERVICE_ACCOUNT} -o json | jq -Mr '.secrets[].name | select(contains("token"))')

Step 5: Extract Bearer token from Secret and decode it

TOKEN=$(kubectl get secret ${SECRET} -o json | jq -Mr '.data.token' | base64 -d)

Step 6: Extract, decode and write the ca.crt to a temporary location

kubectl get secret ${SECRET} -o json | jq -Mr '.data["ca.crt"]' | base64 -d > /tmp/ca.crt

Final step: Test access to metrics endpoint

curl -s <API-SERVER>/metrics  --header "Authorization: Bearer $TOKEN" --cacert /tmp/ca.crt | less

Configuring as additional scrape target on Prometheus

Transfer the certificate file from api-server’s VM to Prometheus’ VM. (e.g. destination filename: /opt/api-server-files/ca.crt)

Save the TOKEN obtained on steps above to a file on Prometheus’ VM. (e.g. destination filename: /opt/api-server-files/api-server-token)

Edit Prometheus main configuration file (e.g. /etc/prometheus/prometheus.yml) and add the following scrape target:

  - bearer_token_file: /opt/api-server-files/api-server-token
    job_name: kubernetes-apiservers
    static_configs:
    - targets: ['<API-SERVER-IP>:6443']
    metrics_path: '/metrics'
    scheme: https
    tls_config:
      ca_file: /opt/api-server-files/ca.crt