Kubernetes v1.37: Scale Workloads to Zero with HorizontalPodAutoscaler

Kubernetes v1.37 introduces native support for horizontal autoscaling workloads down to zero replicas as a Beta feature enabled by default. This integration eliminates the requirement for third-party add-ons or Alpha feature gates, enabling direct control through the HorizontalPodAutoscaler (HPA) API. By scaling to zero, engineering teams can optimize resource consumption for asynchronous workloads—such as batch processors and queue consumers—where maintaining idle Pods reserves expensive compute resources like dedicated CPUs or GPUs.

Operationalizing this capability requires a departure from traditional resource-based metrics. Because CPU and memory metrics rely on active Pods, they cannot provide a signal to scale up once replica counts reach zero. Consequently, HPA configurations must utilize object or external metrics that persist independently of the workload. To implement this, engineers must configure a metrics adapter, such as the Prometheus Adapter, to expose external signals—like queue depth—via the External Metrics API.

The system distinguishes between automated zero-scaling and manual intervention using the ScaledToZero condition:

• Automated Scaling: When the HPA scales a target to zero, it sets ScaledToZero=True, authorizing the controller to continue monitoring the external metric for future scale-up events.
• Manual Pause: If a user manually sets a Deployment to zero replicas without the HPA controller’s condition, the workload remains paused and will not respond to metric-based scaling triggers.

Engineers should note that scaling to zero introduces cold-start latency, as the HPA must detect the metric, schedule new Pods, and initialize the application. This model is most effective for event-driven systems with durable buffering layers. HTTP-based applications typically require a separate ingress buffering mechanism, as Kubernetes Services cannot queue requests while no Pods are ready. Prior to implementation, administrators must ensure that both the kube-apiserver and kube-controller-manager are running v1.37 to ensure consistent reconciliation of the ScaledToZero condition across the control plane.

Before configuring a HorizontalPodAutoscaler (HPA) that can scale a workload to zero, understand two prerequisites: the metric source and the API support introduced in Kubernetes v1.37. Traditional CPU or memory metrics disappear when no Pods exist, so the HPA must rely on an object or external metric that remains observable even when the replica count is zero. A common pattern is to use a queue length metric collected by Prometheus.

Install a metrics adapter

The Kubernetes API does not read Prometheus data directly. Deploy a Prometheus Adapter that implements the External Metrics API. Its configuration includes an externalRules entry that maps the Prometheus series queue_consumer_lag to an external metric name:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-adapter-config
data:
  config.yaml: |
    externalRules:
    - seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
      resources:
        overrides:
          namespace:
            resource: namespace

After applying the ConfigMap and the adapter deployment, verify connectivity:

kubectl get --raw '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

If the request returns a numeric value, the metric pipeline is functional and the HPA can be created.

HPA manifest with minReplicas: 0

The following manifest targets a Deployment named queue-worker. It scales between zero and ten replicas, adding one replica for every 30 queued tasks:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
  annotations:
    kubernetes.io/description: "Scales queue-worker based on the number of queued tasks"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: queue_consumer_lag
        selector:
          matchLabels:
            name: worker_tasks
      target:
        type: Value
        value: "30"

Key points to remember:

• The HPA requires at least one external or object metric; a pure CPU/memory HPA will be rejected.
• When the metric reports zero, the controller sets the ScaledToZero=True condition and reduces the Deployment to zero replicas.
• If an operator manually scales the Deployment to zero, the condition remains ScaledToZero=False, keeping the workload paused until a manual scale‑up.
• The default downscale stabilization window is five minutes; adjust spec.behavior.scaleDown if a shorter or longer window is needed.

Deploy the manifest, monitor it with kubectl describe hpa queue-worker, and ensure the external metric remains available. With the beta feature gate enabled by default in v1.37, this configuration provides a reliable “scale‑to‑zero” pattern for batch processors, queue consumers, and other workloads that can tolerate cold‑start latency.

In Kubernetes environments, scaling a workload to zero replicas introduces an ambiguity: a replica count of zero might indicate an automated action performed by the HorizontalPodAutoscaler (HPA) or a manual intervention by an operator intended to pause the deployment. The HPA controller distinguishes these states by utilizing the ScaledToZero status condition, ensuring it only attempts to wake workloads that it explicitly scaled down.

When the HPA controller reduces a deployment to zero replicas, it sets the ScaledToZero condition to True within the HPA status. This record notifies the reconciliation loop that the controller maintains ownership of the state and is authorized to monitor external or object metrics to scale the workload back up when necessary. Conversely, if a deployment reaches zero replicas through manual modification, the ScaledToZero condition remains False, signaling the controller to remain idle and respect the manual pause.

To verify the current state of an HPA and confirm whether the controller is actively managing the zero-replica state, use the following command:

• Execute kubectl describe hpa <hpa-name> to inspect the Conditions section of the output.
• Look for the ScaledToZero condition type.
• If Status is True, the HPA is actively managing the zero state and will continue to evaluate metrics.
• If the condition is False (or absent) while the target replica count is zero, the HPA considers the workload manually paused and will not trigger a scale-up.

It is critical to note that if the underlying metrics adapter fails to report the required metrics, the HPA will report ScalingActive=False. In such scenarios, the HPA cannot scale the workload back up, regardless of the ScaledToZero status. Engineers should monitor for these failures to prevent unexpected downtime, particularly when managing queue-driven workloads that rely on external metric signals.

Operating Kubernetes clusters with version-skewed control planes requires strict synchronization, particularly when utilizing features such as Horizontal Pod Autoscaler (HPA) scaling to zero. In Kubernetes v1.37 and later, the HPAScaleToZero feature is enabled by default. This allows the HPA to scale workloads down to zero replicas and back up, provided the metrics infrastructure is correctly configured.

When executing control plane upgrades, ensure that both the kube-apiserver and kube-controller-manager support and have the feature enabled. If a controller manager running an older version processes an HPA with minReplicas: 0, it will treat the workload as manually paused, preventing the automated scale-up process. Before any rollback or feature gate deactivation, you must manually scale all affected workloads to at least one replica and update the HPA configuration to minReplicas: 1.

For workloads to scale to zero, you must rely on object or external metrics rather than standard resource metrics (CPU or memory). Because standard resource metrics are derived from running pods, they cease to exist once the replica count hits zero, leaving the HPA without a signal to trigger a scale-up. External metrics, such as queue depth, persist independently of the workload, allowing the HPA to evaluate scaling needs even when no pods are active.

Operational stability depends on carefully managing the HPA lifecycle:

• Metrics Validation: Always verify connectivity to the Metrics API using kubectl get --raw before applying HPA configurations.
• Stabilization Windows: Use spec.behavior.scaleDown to configure stabilization windows. The default five-minute window is critical to prevent rapid thrashing due to transient fluctuations in queue length.
• Operational Ambiguity: The ScaledToZero condition distinguishes automatic scale-downs from manual operator pauses. Use kubectl describe hpa to monitor these conditions; if ScaledToZero=False persists without manual intervention, the workload remains paused.
• Readiness: Ensure that your infrastructure includes a metrics adapter, such as the Prometheus Adapter, to translate external data into the Kubernetes External Metrics API.