Kubernetes v1.37 Scheduler Preemption for In‑Place Resize (Alpha)

In‑place pod resize graduated to General Availability in Kubernetes v1.35, allowing containers to change their CPU and memory limits while remaining running. The feature relies on the kubelet to decide whether a resize can be applied immediately or must be postponed.

How the kubelet evaluates a resize request

When a controller (e.g., the Vertical Pod Autoscaler) or a user updates resources.requests for a running container, the kubelet performs the following checks:

• Node capacity check: it compares the requested increase against the node’s current allocatable resources.
• Physical limits: it verifies that the request does not exceed the machine’s total CPU or memory.
• Namespace constraints: it ensures the request respects any LimitRange or admission quota.

If the node has sufficient spare capacity, the kubelet updates the container’s cgroup limits and reports the new values in status.containerStatuses[].resources. If the node cannot satisfy the increase, the kubelet does not reject the request outright; instead it records a resizeStatus of Deferred on the pod.

Deferred vs. Infeasible resize status

A Deferred status means the request is valid but cannot be acted on at the moment because the node’s free resources are insufficient. The pod remains on the node, waiting for other workloads to release capacity or for a preemption event to create headroom. By contrast, an Infeasible status is set when the request violates hard limits such as physical machine capacity, namespace LimitRange, or quota; the kubelet rejects the request immediately and the pod’s spec is left unchanged.

Practical example

# Assume a pod is running on a node with 4 CPU allocatable
kubectl patch pod my-app -p '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"6"}}}]}}' --type=merge
# kubelet response:
# - resizeStatus: Deferred   (node lacks 2 CPU now)
# - resizeStatus: Infeasible (if the node only has 8 CPU total and the request exceeds that)

Implications for operators

• Deferred resizes can become permanently blocked on a fully‑utilized node unless additional capacity is created (e.g., via preemption or node scaling).
• Understanding the distinction helps operators decide whether to manually evict lower‑priority pods, rely on the cluster autoscaler, or enable the upcoming scheduler preemption feature (alpha in v1.37) that automatically clears space for high‑priority Deferred resizes.

When a running Pod requests a vertical scale‑up that exceeds the node’s allocatable headroom, the Kubelet marks the request with a Deferred status. Unlike an infeasible request, a Deferred resize is valid but cannot be actuated until the node frees enough resources. Because the scheduler traditionally treats Pods with spec.nodeName as already placed, it does not revisit these Pods, leaving the resize blocked indefinitely on a fully utilized node.

Before the introduction of scheduler preemption for in‑place Pod resize (available from Kubernetes v1.37 behind the InPlacePodVerticalScalingSchedulerPreemption feature gate), administrators faced a narrow set of remediation steps:

• Manual eviction: Identify lower‑priority Pods on the node and delete or evict them to create headroom.
• Cluster‑autoscaler reliance: Allow the autoscaler to provision a larger node and migrate the Pod, which incurs a restart and defeats the “no‑restart” advantage of in‑place scaling.
• Custom autoscaling logic: Deploy bespoke controllers that trigger node‑size adjustments, adding operational complexity.

These options are disruptive, require human intervention, or depend on external tooling that does not guarantee timely resource release. The operational gap becomes critical when high‑priority workloads—such as an in‑memory database or a real‑time web service—need additional memory to avoid an out‑of‑memory (OOM) crash. If the node is saturated with batch jobs or best‑effort tasks, the critical Pod remains in the Deferred state, risking service degradation or failure.

The impact on production environments includes:

• Reduced ability to bin‑pack low‑priority workloads, forcing clusters to run with idle buffer capacity.
• Increased mean‑time‑to‑recovery (MTTR) for memory‑spike events because the resize cannot proceed automatically.
• Potential violation of service‑level objectives (SLOs) when critical applications cannot scale on demand.

By enabling scheduler‑driven preemption, the kube‑scheduler actively evicts eligible lower‑priority Pods on the same node, freeing the necessary resources for the Deferred resize. This centralizes preemption logic, respects global priorities and Pod Disruption Budgets, and restores the intended “no‑restart” behavior of in‑place vertical scaling while preserving high cluster utilization.

apiVersion: v1
kind: Node
metadata:
  name: batch‑workload‑node
spec:
  podPreemptionPolicy:
    disableResizePreemption:
    - "cluster-autoscaler.kubernetes.io/disable-preemption"
    - "operator.example.com/policy-override"

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  InPlacePodVerticalScalingSchedulerPreemption: true

# Create the cluster
kind create cluster --config kind-config.yaml --image kindest/node:v1.37.0

# Define a node that disables resize preemption
cat > batch-node.yaml <<EOF
apiVersion: v1
kind: Node
metadata:
  name: batch-workload-node
spec:
  podPreemptionPolicy:
    disableResizePreemption:
    - "cluster-autoscaler.kubernetes.io/disable-preemption"
    - "operator.example.com/policy-override"
EOF
kubectl apply -f batch-node.yaml

In‑place pod resizing lets a running container change its CPU or memory limits without a restart, but the kubelet can only apply the new limits when the node has enough allocatable resources. If the node is fully utilized, the kubelet marks the request as Deferred, meaning the resize is valid but waiting for capacity. Prior to Kubernetes v1.37 this state could remain indefinitely because the scheduler did not consider running pods for preemption.

The InPlacePodVerticalScalingSchedulerPreemption feature gate (alpha in v1.37) closes this gap. When enabled, the scheduler watches for pods with a Deferred resize condition and treats the requested resources as already consumed. It then attempts a single‑node preemption: lower‑priority pods on the same node are evicted, freeing the headroom needed for the higher‑priority pod’s resize. This approach preserves the centralised preemption logic of the scheduler, respects Pod Disruption Budgets, and avoids race conditions that could arise if the kubelet performed local preemption.

Enabling the feature gate

• Confirm the cluster runs Kubernetes v1.37 or newer on every control‑plane component and every worker node.
• Add the flag --feature-gates=InPlacePodVerticalScalingSchedulerPreemption=true to kube-apiserver, kube-scheduler, and the kubelet service.
• Restart the components to apply the change.

Disabling resize preemption on specific nodes

Administrators may want to prevent automatic preemption on nodes that are managed by custom autoscaling logic. This is done with the spec.podPreemptionPolicy field on the Node object:

apiVersion: v1
kind: Node
metadata:
  name: batch-workload-node
spec:
  podPreemptionPolicy:
    disableResizePreemption:
    - "cluster-autoscaler.kubernetes.io/disable-preemption"
    - "operator.example.com/policy-override"

When disableResizePreemption is set, the scheduler will ignore Deferred resize requests on that node, allowing controllers such as the Cluster Autoscaler to handle capacity changes without evicting workloads.

Practical verification steps

1. Create a single‑node kind cluster with the feature gate enabled (see the yaml snippet in the evidence).
2. Deploy a low‑priority batch pod that consumes most of the node’s CPU.
3. Update a high‑priority pod’s resources.requests to exceed the remaining allocatable CPU.
4. Observe the scheduler evict the batch pod, after which the kubelet completes the resize and the pod’s status changes from Deferred to Running.

By following these steps, engineers can safely enable scheduler‑driven preemption for in‑place pod resizing, achieve higher node utilisation, and retain fine‑grained control over where automatic preemption is allowed.

Before you can observe scheduler‑driven preemption for a deferred in‑place resize, understand the two states involved:

• Deferred resize: The Kubelet has accepted a valid resource increase but cannot act because the node’s .status.allocatable CPU or memory is exhausted.
• Preemption: With the InPlacePodVerticalScalingSchedulerPreemption feature gate enabled, the scheduler re‑evaluates the node and may evict lower‑priority Pods to free the required capacity.

Step 1 – Prepare a kind configuration

Create kind-config.yaml that enables the feature gate on both the control‑plane components and the kubelet:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  InPlacePodVerticalScalingSchedulerPreemption: true

Step 2 – Spin up a single‑node cluster

Use a node image that runs Kubernetes v1.37 or later, because the feature is introduced in that release:

kind create cluster \
  --config kind-config.yaml \
  --image kindest/node:v1.37.0

After the cluster starts, verify that the node reports its allocatable CPU:

kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu

Typical output on a local kind node shows a single core (e.g., 1), which is deliberately limited to provoke a resource shortage.

Step 3 – Deploy a low‑priority workload

Run a batch job that consumes the remaining CPU:

kubectl run low‑priority \
  --image=busybox \
  --restart=Never \
  --requests=cpu=900m \
  --priority=1000

Assign a higher priority to the critical Pod that will later request a resize:

kubectl run critical-app \
  --image=nginx \
  --restart=Never \
  --requests=cpu=100m \
  --priority=2000

Step 4 – Trigger a deferred resize

Increase the CPU request of critical-app beyond the node’s free capacity:

kubectl patch pod critical-app -p '{"spec":{"containers":[{"name":"nginx","resources":{"requests":{"cpu":"500m"}}}]}}' --type=merge

The kubelet marks the resize as Deferred because only ~100 mCPU is free.

Step 5 – Observe preemption in action

With the feature gate active, the scheduler scans the node, identifies low‑priority as a victim, and evicts it. After eviction, the node’s allocatable CPU rises, allowing the kubelet to complete the resize. Verify the outcome:

kubectl get pod critical-app -o yaml | grep -A2 resizeStatus
kubectl get pod low-priority

You should see resizeStatus: Completed for the critical Pod and the low‑priority Pod terminated. This demonstrates how scheduler‑driven preemption resolves a deferred in‑place resize without manual intervention or node‑level autoscaling.