Optimizing Linux Memory Management: Implementing Kubernetes v1.37 Memory QoS

The progression of Memory Quality of Service (QoS) from its Alpha introduction in Kubernetes v1.22 to its Beta status in v1.37 marks a critical transition in how the kubelet interacts with the Linux kernel's cgroup v2 controller. By shifting memory management from user-space pressure stall information (PSI) heuristics to kernel-native mechanisms, Kubernetes can now enforce memory constraints with significantly higher precision.

A core architectural challenge in this evolution was ensuring backward compatibility during the shift to enabling the feature gate by default. In previous Alpha releases, memoryThrottlingFactor defaulted to 0.9, which triggered automatic memory.high settings. To prevent unexpected workload throttling during node upgrades in v1.37, the default was changed to null. Consequently, the kubelet no longer writes memory.high, memory.min, or memory.low values to cgroups unless explicitly configured, ensuring that existing runtime behaviors remain idempotent unless the operator explicitly opts in.

The transition introduces two primary control levers for cluster operators to manage memory contention:

• memoryThrottlingFactor: When set (e.g., 0.9), the kubelet calculates and applies memory.high limits to Burstable and BestEffort containers.
• memoryReservationPolicy: When set to TieredReservation, the kubelet enforces memory.min for Guaranteed pods and memory.low for Burstable pods, providing a mechanism for memory protection.

Example configuration for enabling tiered reservation:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9
memoryReservationPolicy: TieredReservation

Engineers must be aware that the current implementation of TieredReservation is node-wide; it applies to every pod on the host, which may create conflicts when mixing latency-sensitive workloads with high-churn, I/O-heavy jobs. Because these protections also account for page cache, misconfigured policies can unintentionally pin large amounts of memory, reducing the kernel's ability to reclaim resources for neighboring processes. Future iterations are expected to address this granularity, but current operational strategy requires careful alignment of pod resource requests with kernel-level memory reclamation policies.

In Kubernetes v1.37 the MemoryQoS feature gate graduates to Beta and is enabled by default on every kubelet. The gate itself does not impose any cgroup limits unless the kubelet configuration explicitly sets memoryThrottlingFactor or memoryReservationPolicy. Previously, the Alpha release defaulted memoryThrottlingFactor to 0.9, which caused the kubelet to write a memory.high value for Burstable and BestEffort pods as soon as the gate was turned on.

During a cluster upgrade, the automatic insertion of memory.high would have altered the runtime behavior of existing workloads that were never throttled. This unintended throttling could manifest as sudden OOM‑kill events or degraded latency for latency‑sensitive services that rely on the kernel’s reclaim path. To preserve backward compatibility, the default was changed to null, meaning “do not set memory.high unless the user opts in.”

• Safety first: A null default guarantees that upgrading to v1.37 does not modify cgroup settings for pods that lack an explicit factor.
• Explicit opt‑in: Administrators who require throttling must add memoryThrottlingFactor: 0.9 (or another value between 0 and 1) to the KubeletConfiguration file.
• Configuration preservation: If a cluster already contains a non‑null factor, the upgrade process retains that value, so existing throttling behavior continues unchanged.

Example of an explicit opt‑in configuration:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

When the factor remains null, the kubelet skips writing memory.high, and the kernel treats container memory as fully reclaimable, matching the pre‑v1.37 semantics. This design trade‑off sacrifices automatic protection for Burstable/BestEffort pods in favor of a predictable upgrade path. Operators who need tiered reservation can still enable it independently by setting memoryReservationPolicy: TieredReservation, which also defaults to None to avoid unintended node‑wide memory reservations.

The TieredReservation policy is activated by setting memoryReservationPolicy: TieredReservation in the kubelet configuration. When the policy is in effect on a node that runs cgroup v2, the kubelet writes two distinct cgroup limits:

• memory.min on every Guaranteed pod – a hard reservation that the kernel will never reclaim, even under extreme memory pressure.
• memory.low on every Burstable pod – a soft limit that the kernel may reclaim only after all memory.min reservations have been satisfied.

This tiered approach protects the memory guarantees of high‑QoS workloads while still allowing the node to reclaim memory from lower‑QoS containers when needed.

Typical kubelet configuration enabling both throttling and tiered reservation looks like:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9          # optional, enables memory.high
memoryReservationPolicy: TieredReservation

When memoryThrottlingFactor is omitted (the default null in v1.37), the kubelet does not set memory.high, so only the reservation limits are applied. The policy is node‑wide; there is no per‑pod opt‑out. Consequently, every Guaranteed pod receives a memory.min equal to its requests.memory, and every Burstable pod receives a memory.low equal to its requests.memory multiplied by the memoryThrottlingFactor (if throttling is enabled).

Key trade‑offs to consider:

• Hard reservation scope: memory.min covers all memory charged to the pod’s cgroup, including page cache. A pod that buffers large files can therefore consume node memory that would otherwise be reclaimable for its peers.
• All‑or‑nothing policy: Because the reservation policy applies to the entire node, clusters that mix workloads requiring strict reservation with workloads that should remain fully reclaimable must choose a single policy or run separate node pools.
• Eviction behavior: Under memory pressure, the kernel first evicts containers that have only memory.low set, then proceeds to containers without any reservation. Guaranteed pods with memory.min are only evicted as a last resort, effectively shielding them from OOM‑kill.

Operators should monitor node memory usage after enabling TieredReservation, especially for workloads with heavy file I/O, and adjust pod memory requests accordingly to avoid unintentionally exhausting node capacity.

In Kubernetes v1.37 the MemoryQoS feature gate graduates to Beta and is enabled by default on nodes that run cgroup v2. The default kubelet configuration deliberately leaves both memoryThrottlingFactor and memoryReservationPolicy unset, which means no memory.high, memory.min, or memory.low values are written to the cgroup hierarchy unless an operator explicitly opts in.

To activate memory throttling only, set a factor between 0 and 1. The kubelet multiplies the container’s request by this factor to compute memory.high for Burstable and BestEffort pods. A typical 0.9 factor reserves 90 % of the request as a hard limit, allowing the kernel to reclaim the remaining 10 % under pressure.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

Tiered reservation introduces two additional cgroup controls: memory.min for Guaranteed pods and memory.low for Burstable pods. The policy is node‑wide; every pod inherits the same reservation behavior, which simplifies implementation but removes per‑pod granularity.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryReservationPolicy: TieredReservation

When both mechanisms are required, combine the fields. The kubelet will write memory.high based on the throttling factor and simultaneously enforce memory.min/memory.low according to the tiered policy.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9
memoryReservationPolicy: TieredReservation

• Trade‑off – predictability vs. flexibility: Tiered reservation guarantees a minimum memory footprint for Guaranteed pods, reducing OOM risk, but it also reserves page cache, potentially starving neighboring workloads.
• Trade‑off – global policy: Because the reservation policy applies to the entire node, mixed‑workload clusters must choose between hard reservation for all pods or a fully reclaimable pool.
• Upgrade safety: In v1.37 the default memoryThrottlingFactor is null. Clusters that upgrade without modifying their config retain existing behavior; only explicit settings trigger throttling or reservation.

To disable MemoryQoS after an upgrade, clear the feature gate and remove any non‑default fields; the kubelet will reset stale cgroup values to zero on startup.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  MemoryQoS: false

The transition of Memory QoS to Beta in Kubernetes v1.37 introduces nuanced operational constraints, specifically regarding the node-wide scope of memory reservation policies. When memoryReservationPolicy is set to TieredReservation, the kubelet applies memory.min and memory.low settings globally across all pods on the node. This architecture lacks granular, per-pod opt-in mechanisms, creating a rigid enforcement model that may conflict with mixed-workload requirements.

Because these reservations apply at the cgroup level, they inherently encapsulate all memory usage charged to the container, including the page cache. This creates significant trade-offs for I/O-intensive applications. Specifically:

• Page Cache Retention: Pods with large file-system interactions may hold memory buffers that the Linux kernel would otherwise reclaim under pressure, potentially starving neighboring workloads of cache-friendly operations.
• Policy Homogeneity: Since reservations apply to every pod, administrators cannot selectively isolate critical services from reclaimable best-effort workloads. A node must strictly adhere to a single reservation strategy.
• Reclamation Interference: Hard memory reservations effectively bypass standard kernel reclamation logic for the protected amounts, which can lead to OOM conditions if the aggregate reservation exceeds available physical RAM.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryReservationPolicy: TieredReservation 
# Note: Applies memory.min/low to all pods on the node

These architectural limitations—specifically the inability to distinguish between working-set memory and page cache usage during reservation—are primary concerns for SIG Node. The community is actively addressing these constraints; engineers encountering sub-optimal memory pressure scenarios or resource contention due to this global scope are encouraged to contribute their findings to the tracking issue in kubernetes/kubernetes#140246. Until these mechanisms evolve, operators should monitor cache-heavy workloads closely when enabling tiered reservations to prevent unintended node-wide performance degradation.

Decommissioning the Memory QoS feature requires a precise sequence of operations to ensure the kernel cgroup v2 controller returns to a passive state. Because Memory QoS modifies underlying cgroup v2 interfaces—specifically memory.min, memory.low, and memory.high—simply toggling the feature gate may leave stale values persisted in the filesystem hierarchy if not handled by the Kubelet during reconciliation.

To safely disable the feature, follow these architectural requirements:

• Feature Gate Configuration: Explicitly set MemoryQoS: false within the KubeletConfiguration. This informs the Kubelet to cease active management of memory controller interfaces.
• Cleanup of KubeletConfiguration: You must strip any defined memory-specific parameters before applying the new configuration. The Kubelet will reject the configuration if memoryThrottlingFactor is present or if memoryReservationPolicy remains set to TieredReservation.
• Stale Value Reset: Upon service restart with the feature disabled, the Kubelet triggers a reconciliation loop. On cgroup v2 nodes, it automatically resets stale protection values: memory.min=0 and memory.low=0 for the kubepods root cgroup, and memory.low=0 for the Burstable QoS cgroup.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  MemoryQoS: false
# Ensure these fields are removed to avoid validation errors
# memoryThrottlingFactor: 0.9 
# memoryReservationPolicy: None

Note that for container-level cgroups, memory.high values are reset to max (the kernel default) only during the next reconciliation event, such as a container restart or a resource resize operation. If a node is under extreme memory pressure immediately following the disabling of the feature, verify that the kernel has successfully flushed these stale limits to ensure the workload regains access to the full available node memory capacity.