Kubernetes/Requests and Limits, units

Limits and requests

CPU - Meaning of CPU

Limits and requests for CPU resources are measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers and 1 hyperthread on bare-metal Intel processors.

Memory

Limits and requests for memory are measured in bytes. You can express memory as a plain integer (bytes) or as a fixed-point number using one of these suffixes: E, P, T, G, M, k, m (millis). You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

For example, the following represent roughly the same value: 128974848, 129e6, 129M, 128974848000m, 123Mi

CPU units

The CPU resource is measured in CPU units. 1 CPU, in Kubernetes, is equivalent to:

• 1 vCPU/Core for cloud providers (1 AWS vCPU, 1 GCP Core, 1 Azure vCore)
• 1 Hyperthread on a bare-metal Intel processor with Hyperthreading

Fractional values are allowed. A Container that requests 0.5 CPU is guaranteed half as much CPU as a Container that requests 1 CPU. You can use the suffix m to mean milli. For example 100m CPU, 100 milliCPU, and 0.1 CPU are all the same. Precision finer than 1m is not allowed.

CPU is always requested as an absolute quantity, never as a relative quantity; 0.1 is the same amount of CPU on a single-core, dual-core, or 48-core machine.

Memory Units

Fixed-Point Suffix

• E = Exabyte. 1E = 1,000,000,000,000,000,000 bytes
• P = Petabyte. 1P = 1,000,000,000,000,000 bytes
• T = Terabyte. 1T = 1,000,000,000,000 bytes
• G = Gigabyte. 1G = 1,000,000,000 bytes
• M = Megabyte. 1M = 1,000,000 bytes
• K = Kilobyte. 1K = 1,000 bytes

Note: Use it if you’re mostly used to seeing the units of, files on your computer drives; they just exclude B in the suffix.

Binary, Power-of-Two Suffix

• Ei = EiB = Exbibyte. 1Ei = 2⁶⁰ = 1,152,921,504,606,846,976 bytes
• Pi = PiB = Pebibyte. 1Pi = 2⁵⁰ = 1,125,899,906,842,624 bytes
• Ti = TiB = Tebibyte. 1Ti = 2⁴⁰ = 1,099,511,627,776 bytes
• Gi = GiB = Gibibyte. 1Gi = 2³⁰ = 1,073,741,824 bytes
• Mi = MiB = Mebibyte. 1Mi = 2²⁰ = 1,048,576 bytes
• Ki = KiB = Kibibyte. 1Ki = 2¹⁰ = 1,024 bytes

Note: Both the suffixes with and without the trailing B work for the binary, power-of-two suffixes.

Compare

128974848≈129e6≈129M       # Fixed-point (metric)
128974848≈129e6≈129M≈123Mi # Binary

Set requests and limits

• PromQL sources

Find the containers without Kubernetes resource limits

# Containers without CPU limits by namespace 
sum by (namespace)(count by (namespace,pod,container)(kube_pod_container_info{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"}))

# Containers without memory limits by namespace
sum by (namespace)(count by (namespace,pod,container)(kube_pod_container_info{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"}))

# Top 10 containers without CPU limits, using more CPU
topk(10,sum by (namespace,pod,container)(rate(container_cpu_usage_seconds_total{container!=""}[5m])) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"}))

# Top 10 containers without memory limits, using more memory
topk(10,sum by (namespace,pod,container)(container_memory_usage_bytes{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"}))

Detecting containers with very tight CPU limits could cause CPU throttling

# CPU usage is close to its limits
(sum by (namespace,pod,container)(rate(container_cpu_usage_seconds_total{container!=""}[5m])) / sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"})) > 0.8

# Memory usage is close to its limits - could cause OOMKilled
(sum by (namespace,pod,container)(container_memory_usage_bytes{container!=""}) / sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"})) > 0.8

How to set the right Kubernetes resource limits?

Conservative - We will select the value of the container that consumed the most in each moment. If we set the limit to that value, the containers won’t run out of resources.

# Finding the right CPU limit, with the conservative strategy
max by (namespace,owner_name,container)((rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[5m])) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))

# Finding the right memory limit, with the conservative strategy
max by (namespace,owner_name,container)((container_memory_usage_bytes{container!="POD",container!=""}) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))

Has the cluster enough capacity? Finding the overcommit of the cluster.

# % memory overcommitted of the cluster
100 * sum(kube_pod_container_resource_limits{container!="",resource="memory"} ) / sum(kube_node_status_capacity_memory_bytes)

# % CPU overcommitted of the cluster
100 * sum(kube_pod_container_resource_limits{container!="",resource="cpu"} ) / sum(kube_node_status_capacity_cpu_cores)

Finding the overcommit of the node

It is also important to check the overcommit per node. An example of node overcommit can be a pod with a request of 2 CPUs and a limit of 8. That pod can be scheduled in a node with 4 cores, but as the pod has 8 cores as the limit, the overcommit in that node would be 8 – 4 = 4 cores.

# % memory overcommitted of the node
sum by (node)(kube_pod_container_resource_limits{container!="",resource="memory"} ) / sum by (node)(kube_node_status_capacity_memory_bytes)

# % CPU overcommitted of the node
sum by (node)(kube_pod_container_resource_limits{container!="",resource="cpu"} ) / sum by (node)(kube_node_status_capacity_cpu_cores)