Kubernetes/Kubelet

The kubelet is the primary “node agent” that runs on each node. The kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy.

Kubelet provides also interface to gether resource metrics or full metrics pipelines to collect monitoring statistics - used by autoscaling, monitoring and health checks solutions.

A few interesting options:

• --port int32 :- the port for the Kubelet to serve on. (default 10250)
• --read-only-port int32 :- the read-only port for the Kubelet to serve on with no authentication/authorization (set to 0 to disable) (default 10255)
• --healthz-port int32 :- the port of the localhost healthz endpoint (set to 0 to disable) (default 10248)
• --max-pods int32 :- number of Pods that can run on this Kubelet. (default 110)

Check what options are applied, ssh to one of the nodes and check kubelet process

kubectl get nodes -owide
NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE              KERNEL-VERSION   CONTAINER-RUNTIME
minikube   Ready    master   45m   v1.15.2   10.0.2.15     <none>        Buildroot 2018.05.3   4.15.0           docker://18.9.8

ps -aux | grep kubelet
root      3409  8.3  4.8 1353596 96020 ?       Ssl  20:54   2:40 /usr/bin/kubelet --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --container-runtime=docker --fail-swap-on=false --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests
root      3819  8.5 14.2 471736 284396 ?       Ssl  20:54   2:43 kube-apiserver --advertise-address=192.168.99.104 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=8443 --service-account-key-file=/var/lib/minikube/certs/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/var/lib/minikube/certs/apiserver.crt --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
docker   18542  0.0  0.0   9240   476 pts/0    S+   21:26   0:00 grep kubelet

Check check ports

# Http
curl http://localhost:10255/stats/summary
{
  "node": {
   "nodeName": "minikube",
   "systemContainers": [
    {
     "name": "runtime",
     "startTime": "2019-08-28T21:14:07Z",
     "cpu": {
      "time": "2019-08-28T21:32:41Z",
      "usageNanoCores": 80916024,
      "usageCoreNanoSeconds": 383367236057
     },
      ...

# Minikube calling kubelet on https
sudo curl https://localhost:10250/stats/summary -k \
  --cert /var/lib/minikube/certs/apiserver-kubelet-client.crt \
  --key  /var/lib/minikube/certs/apiserver-kubelet-client.key

sudo curl https://${HOSTNAME}:10250/stats/summary \
  --cert /var/lib/minikube/certs/apiserver-kubelet-client.crt \
  --key  /var/lib/minikube/certs/apiserver-kubelet-client.key

Troubleshooting

sudo journalctl -u kubelet
sudo more syslog | tail -120 | grep kubelet

# restart service
sudo systemctl enable kubelet && systemctl start kubelet

# If swap is enabled kubelet won't start
sudo su -
swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab

References

• Kubelet K8s docs