Migrating Kubernetes Clusters to cgroup v2: A Technical Migration Guide

cgroup v1 exposed a separate hierarchy for each resource controller (cpu, memory, blkio, etc.). This fragmentation forced the kubelet and container runtimes to mount and manage multiple trees, often leading to mismatched controller placement and inconsistent enforcement across Pods. In contrast, cgroup v2 introduces a single, unified hierarchy where all controllers are attached to the same node in the filesystem. The result is a deterministic path—typically /sys/fs/cgroup—that every container shares, simplifying both discovery and policy application.

From an isolation standpoint, the unified model enables the kernel to apply a coherent set of limits without the “controller‑scatter” bugs that plagued early Kubernetes releases. For example, memory QoS in Kubernetes v1.36 relies on the v2 memory.high throttling knob and the memory.min/memory.low protection levels. These controls cannot be expressed in v1 because the memory controller lived in a separate tree and lacked hard‑/soft‑reservation semantics.

• Consistent interface: All controllers expose a single set of files (e.g., cpu.weight, memory.max, pids.max) instead of the mixed cpu.shares, memory.limit_in_bytes, etc.
• Improved OOM handling: Setting memory.oom.group ensures an OOM event kills the entire container cgroup, avoiding partially functional multi‑process Pods.
• Rootless delegation: v2 officially supports delegating controllers to non‑root users via cgroup.subtree_control, a capability missing from v1.
• Observability: The memory.events and PSI files (cpu.pressure, memory.pressure) provide per‑cgroup contention metrics that the kubelet surfaces through the Summary API.

Migration requires a kernel ≥ 5.8 (5.9+ recommended for memory QoS) and a runtime that understands the v2 driver (e.g., containerd v1.4+, CRI‑O v1.20+). Kubernetes v1.35 defaults failCgroupV1: true, preventing the kubelet from starting on a v1 node, and the deprecation path is tracked in KEP‑5573. The following snippet shows a minimal kubelet configuration that opts into the unified hierarchy:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
featureGates:
  MemoryQoS: true
memoryReservationPolicy: TieredReservation

While v2 eliminates many of the legacy inconsistencies, trade‑offs remain: older monitoring tools that read v1 files must be updated, and the non‑linear conversion from cpu.shares to cpu.weight can surprise policies that assume a linear scale. Nonetheless, the unified hierarchy delivers a more predictable, secure, and observable foundation for modern, multi‑tenant workloads.

Successful migration to a modern Kubernetes environment necessitates strict adherence to host kernel and runtime configurations, specifically regarding control group (cgroup) management. As Kubernetes pivots toward cgroup v2 as the primary resource management interface, engineers must ensure the following environment prerequisites are satisfied to avoid runtime failures and ensure stable resource isolation.

Kernel and Runtime Prerequisites:

• Kernel Version: A minimum Linux kernel version of 5.8 is strictly required for baseline cgroup v2 functionality. However, for clusters leveraging Memory QoS features, kernel 5.9+ is strongly recommended to prevent potential livelock conditions during memory pressure events.
• Container Runtime: Compatibility requires containerd v1.4+ or CRI-O v1.20+. Engineers are advised to standardize on containerd v2.0+, which provides native, automatic cgroup-driver discovery, reducing configuration drift.
• Cgroup Driver Parity: The kubelet and the container runtime (CRI) must be configured with identical cgroup drivers. Mismatches—such as the kubelet using cgroupfs while the runtime defaults to systemd—will cause critical node instability and erratic resource accounting.

Configuration Mechanics:

When deploying or upgrading nodes, verify the driver synchronization via the kubelet configuration file. In production environments utilizing systemd as the init system, it is standard practice to force both components to use the systemd driver to ensure proper scope management.

# Example KubeletConfiguration snippet
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Migration Considerations:

With Kubernetes v1.35 and later, failCgroupV1 defaults to true. The kubelet will fail to initialize on nodes still utilizing cgroup v1 unless an explicit override is applied. Furthermore, the system-validators preflight checks implemented in kubeadm will now return fatal errors for v1 configurations. Before upgrading, validate that all monitoring tools, eBPF agents (like Cilium), and OCI runtimes (e.g., runc 1.3.2+) are updated to support the unified cgroup v2 hierarchy, as legacy tools relying on v1-specific interface files will cease to function correctly.

Kubernetes has moved from cgroup v1 to cgroup v2, and the CPU controller reflects that change: the legacy cpu.shares file is replaced by cpu.weight. In cgroup v1 a container’s relative CPU entitlement is expressed as an integer in the range 2–262144, where the default of 1024 represents “normal” priority. cgroup v2 uses a weight range of 1–10000, with 100 as the default. The shift is not a simple linear scaling; the kernel expects a non‑linear mapping to preserve the original priority semantics while giving finer granularity for small requests.

The conversion implemented in recent OCI runtimes follows the formula defined in the kernel documentation:

weight = 1 + ((shares - 2) * 9999) / 262142

For example, a pod that requests cpu.shares: 512 (half of the default) is translated to cpu.weight: 500. A request of cpu.shares: 2 (the minimum) yields cpu.weight: 1, while the default 1024 becomes 100. This non‑linear curve keeps the default priority unchanged and expands the usable range for low‑share containers, which previously suffered from coarse granularity.

Both crun v1.23 and runc v1.3.2 embed this conversion in the OCI runtime layer, not in Kubernetes itself. When the runtime sees a cpu.shares entry in the container’s spec, it computes the corresponding cpu.weight and writes it to the cgroup v2 filesystem before the container starts. The conversion is transparent to the kubelet, but monitoring or policy tools that read cpu.weight directly must be updated to expect the new range.

• Verify the runtime version (≥ crun 1.23 or runc 1.3.2) on each node.
• Update any custom admission controllers or autoscalers that calculate exact cpu.weight values.
• Adjust observability pipelines to interpret the 1‑10000 range instead of 2‑262144.
• Test in‑place vertical scaling (stable since Kubernetes v1.35) to ensure the kubelet correctly propagates weight changes.

Adopting the non‑linear conversion eliminates the “one‑size‑fits‑all” limitation of cpu.shares, but it also introduces a new numeric domain that all tooling must understand to avoid mis‑allocation of CPU resources.

The transition to cgroup v2 provides a unified hierarchy and refined interfaces for resource management, which are prerequisites for modern Kubernetes Memory QoS. Unlike the fragmented cgroup v1 model, cgroup v2 allows for explicit, tiered memory protection mechanisms that enable precise control over container-level memory pressure and eviction behaviors.

Memory QoS in Kubernetes relies on three primary cgroup v2 control files to establish a hierarchy of resource entitlement:

• memory.min: Implements hard protection. When set, the kernel will not reclaim memory from the cgroup unless the cgroup is already exceeding its memory.min allocation. If no further reclaimable memory exists outside of this protection zone, the system may experience OOM (Out-of-Memory) conditions.
• memory.low: Implements soft protection. The kernel prioritizes reclaiming memory from other cgroups before attempting to reclaim from a cgroup under this threshold. This allows for performance stability during transient pressure without the strict enforcement of hard limits.
• memory.high: Implements a throttling mechanism. When a cgroup approaches this threshold, the kernel forces the cgroup to perform aggressive reclaim and throttles the cgroup’s memory allocation rate, effectively acting as a reactive "soft limit."

In Kubernetes v1.36, enabling TieredReservation via KubeletConfiguration maps these cgroup v2 primitives to Pod QoS classes. Guaranteed Pods are assigned memory.min to ensure their requests remain immune to eviction, while Burstable Pods utilize memory.low to define their protection priority. BestEffort Pods receive no protection, remaining the primary candidates for kernel-level reclamation.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  MemoryQoS: true
memoryReservationPolicy: TieredReservation
memoryThrottlingFactor: 0.9

When implementing these controls, engineers must account for the memoryThrottlingFactor, which derives the memory.high limit from the Pod's memory request and limit. Careful tuning is required to prevent "livelock" scenarios, particularly on kernel versions prior to 5.9, where improper memory.high configuration can lead to kernel-level performance degradation under heavy I/O workloads.

Pressure Stall Information (PSI) is a kernel‑level metric that quantifies CPU, memory, and I/O contention per cgroup. On clusters that have migrated to cgroup v2 (Linux 4.20+, CONFIG_PSI=y, and no psi=0 boot flag), the kubelet automatically reads the memory.pressure, cpu.pressure, and io.pressure files under each container’s cgroup directory and surfaces the values through the Summary API and /metrics/cadvisor. A typical monitoring loop looks like:

while read -r line; do
    # Example line: memory.pressure avg10=0.12 avg60=0.08 avg300=0.05 total=12345
    eval $line
    echo "$(date) mem10=$avg10 mem60=$avg60 mem300=$avg300"
done < /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid>.slice/container.slice/memory.pressure

PSI data enables proactive scaling decisions and can be correlated with OOM events to distinguish genuine memory starvation from transient cache pressure.

When a container exceeds its memory.limit_in_bytes, the kernel triggers an OOM event. In cgroup v2 the kubelet defaults singleProcessOOMKill to false, which writes memory.oom.group=1 for each container cgroup. This causes the kernel to kill the entire cgroup atomically, preserving container integrity. Enabling singleProcessOOMKill: true reverts to the legacy cgroup v1 behavior where the kernel may terminate a single offending process, which can leave multi‑process containers partially functional. The setting is scoped per container, not per pod, and can be overridden in the KubeletConfiguration:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
singleProcessOOMKill: true

Updating userspace tools that read the cgroup filesystem directly is essential after migration:

• Replace reads of cpu.shares with cpu.weight (e.g., cat /sys/fs/cgroup/.../cpu.weight).
• Switch from memory.stat fields like active_file to the unified memory.events counters for OOM tracking.
• Upgrade cAdvisor to ≥ 0.43.0 and ensure language‑specific libraries (Java, Node.js, automaxprocs) are compiled against the new cgroup layout.

These changes reduce false‑positive OOM kills, improve observability via PSI, and align tooling with the single‑process OOM semantics required for reliable enterprise workloads.

Kubernetes has placed cgroup v1 in maintenance mode as of v1.31 and will reject it by default starting with v1.35, where the failCgroupV1 flag defaults to true. The deprecation lifecycle therefore consists of three phases: (1) detection, (2) temporary override, and (3) enforced removal.

Detection via system‑validators

The k8s.io/system-validators package implements the SystemVerification preflight check. During kubeadm init, kubeadm join, or kubeadm upgrade it scans /proc/self/mountinfo for a mixed or v1 hierarchy and aborts with an error when the kubelet version is ≥ v1.35. Example output:

$ kubeadm upgrade apply v1.35.0
[preflight] Running preflight checks
[preflight] SystemVerification: error: cgroup v1 detected on node "worker-01"

Temporary bridge using failCgroupV1

Administrators can keep a mixed cluster alive by explicitly disabling the fail‑fast behavior in the kubelet configuration. This is intended only as a short‑term migration aid; the flag will be removed following the standard Kubernetes deprecation policy.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false   # temporary override, remove before v1.38

When the flag is set to true (or omitted on v1.35+), the kubelet aborts on start‑up if any node reports a v1 hierarchy, preventing accidental roll‑backs.

Roadmap for full decommissioning

• Upgrade every Linux node to a kernel ≥ 5.8 (5.9+ recommended for MemoryQoS) and enable cgroup v2 at boot (e.g., systemd.unified_cgroup_hierarchy=1).
• Verify container runtime support: containerd v1.4+ or CRI‑O v1.20+ must be configured with --cgroup-driver=systemd and discover the v2 driver automatically.
• Run the system‑validator preflight in --dry-run mode across the fleet to confirm compliance before a production upgrade.
• Remove the failCgroupV1: false override from the KubeletConfiguration and redeploy the kubelet.
• After all nodes report a unified hierarchy, upgrade to Kubernetes v1.36+ to unlock cgroup v2‑only features such as MemoryQoS, PSI, and container‑aware OOM handling.

By treating failCgroupV1 as a controlled gate and leveraging the built‑in system‑validators, enterprises can automate the detection of legacy cgroup usage, enforce a clean cut‑over window, and avoid runtime failures during large‑scale cluster upgrades.