Kubernetes Distributed Tracing (Jaeger)

Table of Contents

Overview

Jaeger is a tool to trace requests within an application distributed over a kubernetes cluster.

It requires to deploy a sidecar container within each pod running the application we want to monitor.

The sidecar container (jaeger-agent) will collect data from the pod and send the same to the jaeger-server (running on a dedicated pod).

Installation

Requirements

• Deploy an ingress-controller

helm upgrade --install ingress-nginx ingress-nginx   --repo https://kubernetes.github.io/ingress-nginx   --namespace ingress-nginx --create-namespa

• Deploy a cert-manager (check latest release available here: https://github.com/cert-manager/cert-manager)

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.6.3/cert-manager.yaml

• Create namespace observability

kubectl create namespace observability

• Deploy observability operator (check latest version available here: https://github.com/jaegertracing/jaeger-operator/tree/main)

kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.36.0/jaeger-operator.yaml -n observability

• Create a resource implementing the operator created on step above

kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
EOF

The following objects must be available in the cluster once installation is successfully completed:

[root@mr-k8s-demo1 ~]# k get all -n observability
NAME                                  READY   STATUS    RESTARTS   AGE
pod/jaeger-operator-f9d5b7569-9px49   2/2     Running   0          2d23h

NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/jaeger-operator-metrics           ClusterIP   10.98.4.211     <none>        8443/TCP   2d23h
service/jaeger-operator-webhook-service   ClusterIP   10.97.240.143   <none>        443/TCP    2d23h

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger-operator   1/1     1            1           2d23h

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/jaeger-operator-f9d5b7569   1         1         1       2d23h

Configuring app to be monitored

The app that needs to be monitored must implement open-tracing tracer initialization.

Implementation differs depending on the language/framework in use.

Jaeger agent, running as sidecar, can be injected by adding the following annotation:

apiVersion:apps/v1
kind:Deployment
metadata:name:test-app
  labels:name:test-app
  annotations:"sidecar.jaegertracing.io/inject":"true"

Accessing jaeger web console

Jaeger server is by default available on port 16686.

For testing / occasional monitoring purposes, the port can be made available using kubernetes’ built-in port-forward:

kubectl port-forward --address 0.0.0.0  $(kubectl get pods -l=app="jaeger" -o name) 16686:16686

From this moment, jaeger web UI is accessible at http://k8s_host_ip_address:16686