Kubernetes/Monitoring

Monitor cluster resources

Metric-server

In order to get cluster resources you need a metric collector plugin. Popular one is heapster that exposes metric-server service. The below commands relay on its API to get data:

Install metrics-server

git clone https://github.com/kubernetes-incubator/metrics-server.git
kubectl apply -f ~/metrics-server/deploy/1.8+/

Get metrics

# verify metrics server API
kubectl get --raw /apis/metrics.k8s.io/

kubectl top node               # CPU,memory utilization of the nodes in your cluster
kubectl top pods               # CPU,memory utilization of the pods in your cluster
kubectl top pods -A            # CPU,memory of pods in all namespaces
kubectl top pod -l run=<label> # CPU and memory of pods with a label selector:
kubectl top pod <pod-name>     # CPU,memory of a specific pod
kubectl top pods group-context --containers # CPU,memory of the containers inside the pod

cAdvisor deprecated v1.11]

Every node in a Kubernetes cluster has a Kubelet process. Within each Kubelet is a cAdvisor process. The cAdvisor is continuously gathering metrics about the state of the cluster. It's always available

minikube start --extra-config=kubelet.CAdvisorPort=4194
kubectl proxy &          # open a proxy to the Kubernetes API port
open $(minikube ip):4194 # cAdvisor also serves up the metrics is a helpful HTML format

# Each node provide statistics that are provided by cAdvisor. Access the node stats
curl localhost:8001/api/v1/nodes/$(kubectl get nodes -o=jsonpath="{.items[0].metadata.name}")/proxy/stats/

# Kubernetes API also gather the cAdvisor metrics at /metrics
curl localhost:8001/metrics

Liveness and Readiness probes

Check this Visual explanation

Get service endpoints

kubectl get endpoint

Logs

Container logs

These are stored on nodes in /var/log/ directory and contain everything containers send to STDOUT.

• /var/log/containers/ contains container logs, these are symlinks to ../pods/
• /var/log/containers/ contains directory per each pod in form <namespace-<rs|deployment>/<pod-name>/0.log(logfile)
• 0.log it's a symlink to /var/lib/docker/containers/uid-part1/uid-part2-json.log

$ ls -l /var/log/containers
total 56
lrwxrwxrwx 1 root root 101 Oct  7 06:51 coredns-5644d7b6d9-hztth_kube-system_coredns-9de9395495186177f5112d795ca950dd0227e6f025f40c83ddf2a99c56802939.log -> /var/log/pods/kube-system_coredns-5644d7b6d9-hztth_5da159b3-64e7-48e4-b9f8-003f9623481d/coredns/0.log
...

# kubelet runs as a process therefore writes logs to system location
/var/log
journalctl -u kubelet.service

Retrieve logs

kubectl logs <pod> <container> # container name is optional for a single container pods
kubectl logs <pod> <container> --previous flag # in case the container has crashed

Resources

• Logging K9s docs