Kubernetes v1.37: Scale Workloads to Zero with HorizontalPodAutoscaler

The HorizontalPodAutoscaler (HPA) in Kubernetes v1.37 adds native support for scaling workloads down to zero replicas. This capability is now a Beta feature, enabled by default in both the kube‑apiserver and kube‑controller‑manager, and no longer requires an external add‑on or an Alpha feature gate.

Why object or external metrics are required

Traditional HPA scaling relies on CPU or memory metrics, which are produced only while Pods are running. When the replica count reaches zero, those metrics disappear, leaving the HPA without a signal to scale back up. Object and external metrics, such as a queue length, exist independently of the worker Pods and can therefore drive both down‑scaling and up‑scaling.

Configuring an external metric

Assume a Prometheus metric queue_consumer_lag that reports the number of pending tasks. The Prometheus Adapter must expose this metric through the External Metrics API. A minimal adapter rule looks like:

externalRules:
- seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
  metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
  resources:
    overrides:
      namespace:
        resource: namespace

Validate the metric exposure with:

kubectl get --raw '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

Example HPA definition

The following HPA scales a Deployment named queue-worker between 0 and 10 replicas, adding one replica for every 30 queued tasks:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: queue_consumer_lag
        selector:
          matchLabels:
            name: worker_tasks
      target:
        type: Value
        value: "30"

Operational considerations

• Cold‑start latency: After scaling to zero, the HPA must observe the metric, schedule a Pod, and start the application before processing can resume.
• Stabilization window: The default downscale window is five minutes; adjust spec.behavior.scaleDown if shorter or longer periods are needed.
• Paused vs. scaled‑to‑zero: The controller sets a ScaledToZero=True condition when it performs the downscale. A manual pause lacks this condition, preventing the HPA from automatically restarting the workload.
• Upgrade path: During a control‑plane version skew, ensure both the API server and controller manager have the feature enabled before creating HPAs with minReplicas: 0. When downgrading, change minReplicas to at least 1 and ensure at least one object or external metric is defined.

Next steps for engineers

Start by exposing a durable queue metric via your monitoring stack, then apply the example HPA to a low‑traffic consumer service. Observe the ScaledToZero condition with kubectl describe hpa <name> to verify that the controller owns the zero state. Adjust the downscale stabilization window as needed for your workload’s tolerance to brief load spikes.

HorizontalPodAutoscaler (HPA) implementations that rely on resource metrics such as CPU or memory can only evaluate those metrics while at least one Pod is running. The metrics are collected from the containers themselves; when the replica count reaches 0, the Pods that expose the metrics no longer exist, so the API server receives no data. Without a signal, the HPA has no basis to decide whether to create new Pods, which makes scaling from zero impossible.

Object and external metrics solve this limitation because they are sourced outside the lifecycle of the workload. An object metric (e.g., a custom resource that tracks pending jobs) or an external metric (e.g., a queue length exposed by Prometheus) continues to be available even when no worker Pods are present. The HPA can therefore observe a persistent signal, compare it against the configured target, and trigger a scale‑up.

• Metric availability at zero replicas: Queue length, Kafka lag, or custom counters exist independently of consumers.
• Deterministic scaling logic: The HPA can compute the desired replica count from the metric value (e.g., one replica per 30 queued tasks) regardless of current pod count.
• Clear ownership of zero state: The ScaledToZero condition records whether the HPA performed the downscale, ensuring that only HPA‑managed zero states are re‑evaluated.

Practical example: a deployment named queue-worker uses the Prometheus Adapter to expose the external metric queue_consumer_lag. The HPA definition sets minReplicas: 0 and specifies a target value of 30. When the queue is empty, the metric reports 0, the HPA reduces the deployment to zero replicas, and the ScaledToZero=True condition is set. As soon as new tasks appear, the metric remains observable; the HPA reads the updated value, calculates the required replica count, and creates the Pods.

This approach is essential for workloads that can tolerate cold‑start latency, such as batch processors or queue consumers, where the cost of idle resources outweighs the start‑up delay. By decoupling scaling signals from the Pods themselves, object and external metrics enable reliable, automated scaling both down to and back up from zero.

queue_consumer_lag{namespace="default",name="worker_tasks"}

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-adapter-config
  namespace: monitoring
data:
  config.yaml: |
    externalRules:
    - seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
      resources:
        overrides:
          namespace:
            resource: namespace

kubectl get --raw '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: queue_consumer_lag
        selector:
          matchLabels:
            name: worker_tasks
      target:
        type: Value
        value: "30"

Before creating a HorizontalPodAutoscaler (HPA) that can scale a workload to zero, understand two core concepts: the metric source and the downscale stabilization window. Traditional CPU or memory metrics disappear when no Pods exist, so an object or external metric is required to keep the HPA active while the replica count is zero. The default downscale stabilization window is five minutes; during this period the controller ignores short‑lived drops in the metric to avoid flapping the replica count.

The following manifest demonstrates a complete HPA definition that targets a Deployment named queue-worker. It permits zero to ten replicas and scales based on a Prometheus‑exposed external metric queue_consumer_lag, assuming a Prometheus Adapter is installed and the metric is reachable through the External Metrics API.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
  annotations:
    kubernetes.io/description: "Scales queue-worker based on the number of queued tasks"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: queue_consumer_lag
        selector:
          matchLabels:
            name: worker_tasks
      target:
        type: Value
        value: "30"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # 5 minutes default

• minReplicas: 0 enables true zero‑scale; the API rejects an HPA that relies solely on CPU or memory metrics.
• External metric remains observable even when no Pods run, allowing the HPA to transition back up when the queue length exceeds the target value.
• ScaledToZero condition distinguishes an automatic scale‑down (condition ScaledToZero=True) from a manual pause, ensuring the controller continues to evaluate the external metric.
• Downscale stabilization (default five minutes) prevents a brief dip in queue_consumer_lag from instantly terminating all workers.
• Adjust the stabilization window via spec.behavior.scaleDown.stabilizationWindowSeconds if your workload tolerates faster down‑scaling.

Validate metric availability before applying the HPA (e.g., kubectl get --raw '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'). If the metric cannot be retrieved, the HPA will report ScalingActive=False with a reason such as FailedGetExternalMetric, and the workload will remain at its current replica count.

Scaling workloads to zero in Kubernetes v1.37 allows for significant infrastructure cost reduction, particularly for deployments that reserve expensive hardware resources like dedicated CPUs or GPUs. By utilizing the minReplicas: 0 configuration within a HorizontalPodAutoscaler (HPA), engineers can eliminate idle resource consumption when no tasks are present. However, this architectural pattern introduces a fundamental operational trade-off: cold-start latency.

When an HPA scales a deployment from zero, the system must perform a sequence of operations before the application can process traffic:

• The HPA must detect the metric change and transition the ScaledToZero status condition.
• The Kubernetes scheduler must find an appropriate node and place the Pod.
• The container runtime must pull images and initialize the application runtime.

Because Kubernetes Services do not inherently buffer incoming requests while a deployment has zero replicas, standard HTTP-driven workloads risk dropped connections or timeout errors during this initialization window. To mitigate these risks, engineers should adopt the following strategies:

• Implement Durable Queuing: Scaling from zero is best suited for asynchronous, queue-driven workloads. When work is held in a durable message broker, it remains preserved during the cold-start interval, allowing the newly scheduled Pod to consume pending tasks once initialized.
• Use External Metrics: HPA cannot scale from zero using native resource metrics (CPU or memory) because those metrics require active Pods. You must configure external metrics—such as queue depth—via a metrics adapter (e.g., Prometheus Adapter) so the HPA can maintain visibility into work demand even when the replica count is zero.
• Manage Stabilization Windows: Leverage spec.behavior.scaleDown to configure stabilization windows. This prevents the HPA from prematurely terminating pods during transient fluctuations in queue length, thereby avoiding frequent "thrashing" and repetitive cold starts.

Before implementing this pattern, ensure your control plane is fully upgraded to v1.37, as earlier versions lack the required ScaledToZero condition necessary to distinguish between an automated scale-down and a manual pause.

The HPAScaleToZero feature gate controls whether the HorizontalPodAutoscaler (HPA) can reduce a workload to minReplicas: 0 and later restore it using object or external metrics. In Kubernetes v1.37 the gate is enabled by default on both the kube-apiserver and kube-controller-manager, allowing the API server to accept minReplicas: 0 and the controller manager to apply the ScaledToZero condition that distinguishes an automatic scale‑down from a manual pause.

During a version‑skewed control‑plane upgrade, the two components may run different Kubernetes releases. The HPA will only behave correctly when **both** the API server and the controller manager support the feature and have the gate enabled. If the controller manager is still on a version where the gate is disabled, a replica count of zero is interpreted as a manual pause, and the workload may remain stopped even when the external metric later indicates demand.

To avoid service disruption when disabling the feature gate or rolling back to a version that lacks the condition‑based implementation, follow these steps:

• Raise minReplicas on affected HPAs. Edit each HPA manifest to set minReplicas: 1 (or higher) and commit the change before the downgrade.
• Scale zeroed workloads. For any Deployment that is currently at zero replicas, manually scale it to at least one replica (e.g., kubectl scale deployment my-app --replicas=1).
• Verify metric availability. Ensure that the object or external metric used by the HPA is still being collected; the API server will reject an HPA that relies solely on CPU or memory when minReplicas: 0 is specified.
• Disable the gate. Update the control‑plane component flags (--feature-gates=HPAScaleToZero=false) and restart the affected services.
• Confirm behavior. Run kubectl describe hpa <name> and check that the ScaledToZero condition is absent and that scaling actions respect the new minReplicas value.

When re‑enabling the feature after a successful upgrade, revert the minReplicas changes only after confirming that both control‑plane components are running a version that includes the beta implementation (v1.37 or later). This ensures that the HPA can safely transition between zero and non‑zero replica counts without misinterpreting manual pauses.