Kubernetes/Scheduling

Default scheduler rules

1. Identify if a node has adequate hardware resources
2. Check if a node is running out of resources. check for memory or disk pressure conditions
3. Check if a pod schedule is scheduled to a node by a name
4. Check if a node has a label matching node selector in a pod spec
5. Check if a pod is requesting to bound to a specific host port and if so, does the node have that port available
6. Check if a pod is requesting a certain type of volume be mounted and if other pods are using the same volume
7. Check if a pod tolerates taints of the node, eg. master nodes is tainted with "noSchedule"
8. Check if a pod or a node affinity rules and checking if scheduling the pod would break these rules
9. If there is more than one node could schedule a pod, the scheduler priorities the nodes and choose the best one. If they have the same priority it chooses in round-robin fashion.

Label nodes

kubectl label node worker1.acme.com share-type=dedicated

YAML for the deployment to include the node affinity rules:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: pref
spec:
  replicas: 5
  template:
    metadata:
      labels:
        app: pref
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution: #all pods,but not current pod on the node
          - weight: 80
            preference:
              matchExpressions:
              - key: availability-zone
                operator: In
                values:
                - zone1
          - weight: 20              #4 time less priority then AZ
            preference:
              matchExpressions:
              - key: share-type     #label key
                operator: In
                values:
                - dedicated         #label value
      containers:
      - args:
        - sleep
        - "999"
        image: busybox:v1.28.4
        name: main

Cluster nodes capacity and resources

Check node's capacity

kubectl describe nodes worker-2.acme.com | grep -A 20 Capacity:

<syntaxhighlightjs lang=yaml>

1. ...(output omitted)...

Capacity:

cpu:                2
ephemeral-storage:  20263528Ki
hugepages-2Mi:      0
memory:             4044936Ki
pods:               110

Allocatable:

cpu:                2
ephemeral-storage:  18674867374
hugepages-2Mi:      0
memory:             3942536Ki
pods:               110

System Info:

Machine ID:                 ******c49b4bed31684a******
System UUID:                ******-D110-CB50-EAA3-*******
Boot ID:                    ****8-be21-45ca-b86c-311a479******
Kernel Version:             4.4.0-1087-aws
OS Image:                   Ubuntu 16.04.6 LTS
Operating System:           linux
Architecture:               amd64
Container Runtime Version:  docker://18.6.1
Kubelet Version:            v1.13.10
Kube-Proxy Version:         v1.13.10

PodCIDR: 10.100.1.0/24 Non-terminated Pods: (3 in total)

 Namespace                  Name                           CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
 ---------                  ----                           ------------  ----------  ---------------  -------------  ---
 default                    busybox                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         18h
 kube-system                kube-flannel-ds-amd64-p7c7m    100m (5%)     100m (5%)   50Mi (2%)        50Mi (2%)      5d9h
 kube-system                kube-proxy-27dbb               0 (0%)        0 (0%)      0 (0%)           0 (0%)         5d9h

Allocated resources:

 (Total limits may be over 100 percent, i.e., overcommitted.)
 Resource           Requests   Limits
 --------           --------   ------
 cpu                100m (5%)  100m (5%)
 memory             50Mi (2%)  50Mi (2%)
 ephemeral-storage  0 (0%)     0 (0%)

1. ...(output omitted)...

</syntaxhighlightjs>

Resource request and limit

resource request

is what a pods is guaranteed to get; it's the amount of resources necessary to run a container; a pod will only be scheduled on a node that can give that resource

resource limit

makes sure that a container never goes above a value specified, they allow to up to the limit then are resticted

• Exceeding a memory limit makes your container process a candidate for oom-killing
• process basically can’t exceed the set cpu quota, and will never get evicted for trying to use more cpu time than allocated. The system enforces the quota at the scheduler so the process just gets throttled at the limit.

Units k8s docs

• memory 64Mi - measures in bytes, it means 64 Mebibytes
• cpu 250m - measured in cores, it means 250 miliCPUs or 0.25 CPU core

Note: 1 MiB = 2²⁰ bytes = 1048576bytes = 1024 kibibytes

Schedule a pod with resources request on the specific node <syntaxhighlightjs lang=yaml> apiVersion: v1 kind: Pod metadata:

 name: resource-pod1

spec:

 nodeSelector:
   kubernetes.io/hostname: "worker-2.acme.com"
 containers:
 - image: busybox
   command: ["dd", "if=/dev/zero", "of=/dev/null"]
   name: budybox-dd
   resources:
     requests:
       cpu: 800m     # mili cores -> 2000m=20% of cpu (large for 2nd deployment)
       memory: 20Mi  # Mb

</syntaxhighlightjs>

Create a pod and watch resource request balance changing

kubectl apply -f resource-pod1.yml
watch -d 'kubectl describe nodes worker-2.acme.com  | grep -A 25 Non-terminated'
Non-terminated Pods: (6 in total)
  Namespace          Name                                CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------          ----                                ------------  ----------  ---------------  -------------  ---
  default            busybox                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         24h
  default            nginx-loadbalancer-86bb844fb7-bl5fs 0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  default         -->resource-pod1<--                    800m (40%)    0 (0%)      20Mi (0%)        0 (0%)         6m7s
  kube-system        kube-flannel-ds-amd64-97hvr         100m (5%)     100m (5%)   50Mi (1%)        50Mi (1%)      14d
  kube-system        kube-proxy-fxl6f                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         14d
  rbac1              test-f57db4bfd-ghshj                0 (0%)        0 (0%)      0 (0%)           0 (0%)         12d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                900m (45%)  100m (5%) # <- resources balance
  memory             70Mi (1%)   50Mi (1%)
  ephemeral-storage  0 (0%)      0 (0%)

Then deploying another pod, requesting 2000mi cpus, will end up with scheduling error when describing the pod

kubectl describe pod pod2-2000mi-cpu
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  44s (x10 over 4m57s)  default-scheduler  0/3 nodes are available: 2 node(s) didn't match node selector, 3 Insufficient cpu.

Limits YAML, Unlike requests, limits can go above total utilisation of CPU and memory. K8s will detect if overcommitted and kill the pod. Be aware Containers within pods are not aware of limits sets to pods, this can be seen from top command within a container. <syntaxhighlightjs lang=yaml> apiVersion: v1 kind: Pod metadata:

 name: pod-limit-resources

spec:

 containers:
 - image: busybox
   command: ["dd", "if=/dev/zero", "of=/dev/null"]
   name: main
   resources:
     limits:
       cpu: 2        #by default requests are eq limits if not specified
       memory: 40Mi  #

</syntaxhighlightjs>

Tains and tolerations

Tains and tolerations are restrictions to what PODs can be scheduled on nodes. By default Pods have no tolerations set up.

Taint effects (what happen to intolerant pods, PODs with a no toleration in .spec.toleratins)

• NoSchedule - it simply won't be scheduled
• PreferNoSchedule - the system will try to avoid placing a pod but it's not warrantied
• NoExecute - won't schedule and already running pods will be evicted if don't tolerate the taint, applies to pods that have been scheduled before taint was applied to a node

# Add taint on a node
kubectl taint nodes <node-name> <key>=<value>:<taint-effect>
kubectl taint nodes node1        type=workers:NoSchedule
                              
# Add toleration to a pod
apiVersion:
kind: Pod
metadata:
  name: worker-pod
spec:
  containers:
  - name: nginx
    image: nginx
  tolerations:
  - key: type
    operator: Equal
    value: workers
    effect: NoSchedule

Deamonset

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created. DaemonSets do not use a scheduler to deploy pods, DS used to ignore nodes taints. Some typical uses of a DaemonSet are:

• running a cluster storage daemon, such as glusterd, ceph, on each node.
• running a logs collection daemon on every node, such as fluentd or logstash.
• running a node monitoring daemon on every node, such as Prometheus Node Exporter, collectd

Monitor events and logs

kubectl get events --all-namespaces
kubectl get events --watch #short -w

#See scheduler logs
kubectl logs [kube_scheduler_pod_name] -n kube-system
tail -f /var/log/kube-scheduler.log #run on control plane node

Resources

• Assigning a Pod to a Node
• Pod and Node Affinity Rules
• Configure Multiple Schedulers (our-own)
• DaemonSet