Kubernetes v1.37: Scale Workloads to Zero with HorizontalPodAutoscaler

Traditional HorizontalPodAutoscaler (HPA) configurations rely on resource metrics such as CPU or memory. These metrics are emitted by the kubelet only while a Pod is running, because they are derived from the operating‑system counters of the container. When the replica count reaches 0, the workload no longer has any active Pods, so the resource metrics source disappears. The HPA therefore loses its only signal and cannot decide to create new Pods, which makes scaling from zero impossible with pure resource metrics.

Object and external metrics solve this limitation because they are decoupled from the lifecycle of the Pods that consume them. An object metric (e.g., the length of a Kafka topic) or an external metric (e.g., a Prometheus series named queue_consumer_lag) exists independently of the workers. The HPA can query the External Metrics API at any time, even when minReplicas: 0 and no Pods are scheduled, and use the returned value to calculate a target replica count.

• Metric availability: The metric remains present in the monitoring system (Prometheus, CloudWatch, etc.) regardless of Pod count.
• Deterministic scaling logic: The HPA can map a concrete value (e.g., 30 queued tasks) to a replica count, as shown in the YAML example below.
• Clear ownership of zero state: Kubernetes v1.37 introduces the ScaledToZero condition, allowing the controller to distinguish an automatic scale‑down from a manual pause.

Example: scaling a queue consumer from zero using an external metric.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: queue_consumer_lag
        selector:
          matchLabels:
            name: worker_tasks
      target:
        type: Value
        value: "30"

In this configuration, when the queue length is zero the HPA reduces the Deployment to 0 replicas. As soon as the external metric reports a non‑zero value, the HPA computes the required replica count and schedules new Pods. Because the metric source is external, the scaling decision is possible even during a cold start, eliminating the need for an additional buffering layer for batch‑oriented workloads.

To enable horizontal autoscaling based on data sources external to the Kubernetes cluster, you must utilize the External Metrics API. Unlike CPU or memory metrics, which rely on Pod-level resource utilization, external metrics—such as queue_consumer_lag—exist independently of worker Pods. This independence allows the HorizontalPodAutoscaler (HPA) to trigger scaling events even when the target workload has been scaled to zero.

The Prometheus Adapter serves as the bridge between your Prometheus instance and the Kubernetes metrics pipeline. To expose a metric, you must define an externalRules block within the adapter configuration. This configuration maps your Prometheus series to the API available to the HPA controller:

• seriesQuery: Defines the selector to identify the relevant Prometheus metric.
• metricsQuery: Specifies the PromQL query used to aggregate the metric, typically using <<.Series>> and <<.LabelMatchers>> variables to ensure proper context mapping.

Example configuration snippet:

externalRules:
  - seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
    metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
    resources:
      overrides:
        namespace:
          resource: namespace

Before proceeding with HPA creation, you must verify that the metrics pipeline is correctly exposing the data to the API server. Using kubectl, you can query the External Metrics API directly to ensure the metric is reachable and returning the expected values:

kubectl get --raw '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

This verification step is mandatory. If the adapter fails to return a valid value, the HPA will report ScalingActive=False with a FailedGetExternalMetric reason. An HPA cannot successfully scale a workload—particularly from zero replicas—if the underlying external metric is unavailable or unreachable. Once verified, you can proceed to define the HPA object, ensuring the metrics spec references the exact metric name and label selectors configured in the adapter.

The Kubernetes v1.37 beta introduces native support for scaling a workload to zero replicas using the HorizontalPodAutoscaler (HPA). To exploit this capability, the HPA must target a metric that remains observable when no Pods are running—typically an object or external metric such as a queue length.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
  annotations:
    kubernetes.io/description: "Scales queue-worker based on the number of queued tasks"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: queue_consumer_lag
        selector:
          matchLabels:
            name: worker_tasks
      target:
        type: Value
        value: "30"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # default 5 minutes

Key elements of the manifest:

• minReplicas: 0 enables the controller to reduce the Deployment to zero when the external metric reports no pending work.
• maxReplicas: 10 caps the upward scaling to avoid over‑provisioning.
• The External metric queue_consumer_lag is read via a metrics adapter (e.g., Prometheus Adapter) and remains available even when no Pods exist.
• The behavior.scaleDown.stabilizationWindowSeconds defaults to 300 seconds (five minutes). This window prevents a brief dip in queue length from instantly terminating all workers.

In Kubernetes v1.37, scaling workloads to zero via the HorizontalPodAutoscaler (HPA) is a Beta feature enabled by default. This capability allows deployments—such as queue consumers or batch processors—to scale down to zero replicas, eliminating resource consumption for idle tasks. However, implementing this requires specific architectural considerations regarding metric sources and controller state management.

Operational Requirements and Metrics:

• Metric Dependency: HPA scaling to zero is restricted to object or external metrics (e.g., queue length). Resource metrics like CPU or memory usage are insufficient because they rely on active pods; when the replica count hits zero, these metrics vanish, preventing the HPA from triggering a scale-up.
• Request-Driven Workloads: Kubernetes services do not natively buffer traffic for workloads at zero replicas. Consequently, HTTP-based services require an external buffering layer to accommodate cold-start latency while the HPA schedules and initializes a new pod.
• Validation: Before configuring an HPA with minReplicas: 0, verify the external metric via kubectl get --raw to ensure the metrics adapter is correctly exposing the target value.

Distinguishing Automatic Scale-Down from Manual Pauses:

The controller manages the ambiguity of a zero-replica state using the ScaledToZero status condition. When the HPA automatically reduces a workload to zero, it sets ScaledToZero=True, signaling that it retains ownership of the workload and will continue monitoring metrics. If a user manually scales a deployment to zero, this condition remains false or unset, and the HPA will not attempt to wake the workload. You can inspect this status using kubectl describe hpa <name>.

Upgrade and Rollback Guidance:

During upgrades or rollbacks, maintain stability by adhering to these procedures:

• Control Plane Skew: Ensure both kube-apiserver and kube-controller-manager support the feature before defining minReplicas: 0, as older controllers may interpret zero as a manual pause.
• Downgrade Path: If reverting to a version without integrated support, first update all affected HPAs to minReplicas: 1 and manually scale the target deployments to ensure they are active.