Building a Custom Metrics Exporter for Kubernetes: A Complete Walkthrough

An exporter is a small HTTP server with a single responsibility: exposing application state as plain text on a /metrics endpoint. Prometheus scrapes that endpoint on a regular interval, stores the resulting samples as time-series data, and makes them available for queries, alerts, and autoscaling rules such as the Kubernetes HorizontalPodAutoscaler.

The format Prometheus expects is plain text — one metric per line, consisting of a metric name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the appropriate function when that value changes. A job processor, for example, might expose worker_jobs_processed_total, worker_queue_depth, and worker_job_duration_seconds.

There are two ways to produce these metrics:

• Direct application instrumentation embeds the Prometheus client library inside your application process and exposes /metrics from within it. This works well when you control the application code and the metric values originate inside that process.
• A standalone exporter runs as a separate HTTP server that reads from an external data source and exposes the results. It is preferable when the data source is external to your application or when you do not control the application code, for example with a managed message broker, a database, or a third-party API.

The Prometheus data model defines three main metric types:

• Counters only ever increase and represent totals such as requests served, jobs processed, or errors encountered. Never use a counter for a value that can go down.
• Gauges represent a current snapshot of a value that can rise and fall freely, such as queue depth, active connections, or cache size.
• Histograms record the distribution of observed values, such as request latency, enabling percentile calculations (p99, p50) rather than simple averages.

An exporter that polls its data source should refresh values faster than Prometheus's scrape interval — typically fifteen seconds in most cluster deployments — so each scrape observes a current value. Alongside /metrics, a /healthz endpoint can serve as a liveness probe target without exposing metric data on the health route.

Before instrumenting any application, classify the signal you are measuring. The Prometheus data model defines three primary metric types, and selecting the correct one determines both what the data represents and which queries are valid against it.

• Counters accumulate monotonically. They are the correct representation for totals: requests served, jobs processed, errors encountered. A counter only increases and resets when the process restarts. It is never the right type for a value that can go down, such as the current number of items in memory.
• Gauges represent a current snapshot of a value that can rise and fall freely. Queue depth, active connections, and cache size are all gauges. A gauge reflects state at scrape time and is appropriate when you need to observe the latest value rather than a cumulative total.
• Histograms record the distribution of observed values, such as request latency or job duration. They group observations into configurable buckets, which allows percentile calculations like p50 and p99 to be derived from the stored data, not just averages.

After selecting the metric type, choose a name that follows the convention <namespace>_<name>_<unit> in snake_case. For a job processor, the metrics would be worker_jobs_processed_total (a counter), worker_queue_depth (a gauge), and worker_job_duration_seconds (a histogram). Using a unit suffix such as _total for counters or _seconds for durations makes the meaning explicit and reduces ambiguity for anyone later writing PromQL queries, alerts, or dashboard panels.

Concretely, a worker application would increment the worker_jobs_processed_total counter after each job completes, optionally partitioned by a status label such as success or error. The same application would set worker_queue_depth to the current number of messages waiting in the broker, and call Observe on worker_job_duration_seconds with the measured processing time. These three types cover the majority of application signals and map cleanly to autoscaling decisions, alerting rules, and capacity planning.

To begin instrumenting a Go application for Prometheus, initialize your project as a Go module and fetch the necessary client libraries. Execute the following commands in your project directory:

• go mod init example.com/my-exporter
• go get github.com/prometheus/client_golang/prometheus
• go get github.com/prometheus/client_golang/prometheus/promhttp

The client_golang library provides the primary interface for defining metrics. When declaring these metrics in main.go, use specific types based on the nature of the data:

• CounterVec: Used for cumulative metrics that only increase, such as total requests, utilizing NewCounterVec to support labels.
• Gauge: Represents values that fluctuate, such as active connections, declared via NewGauge.
• Histogram: Tracks distributions of observed values like latency, implemented with NewHistogram and configurable Buckets.

Every metric declaration requires a Help string, which provides essential context for operators viewing the exported data. Once defined, you must register these metrics with the Prometheus registry to ensure they are exposed on the /metrics endpoint. The standard registration pattern is prometheus.MustRegister(), which is ideal for top-level application code because it triggers a panic if a registration fails due to a duplicate metric name, ensuring configuration errors are caught immediately during startup.

However, when embedding an exporter inside a third-party library, avoid MustRegister to prevent unintended application crashes. Instead, use prometheus.Register() and implement manual error handling to check for registration conflicts gracefully. This approach ensures your library remains robust even when integrated into complex systems where multiple components might inadvertently define similar metrics.

Registration effectively informs the Prometheus registry of the metric's existence, allowing it to be serialized into the expected plain-text format when the promhttp handler is invoked by a scrape request.

To ensure observability data remains accurate and timely, you must decouple the metric collection process from the HTTP request cycle. This is best achieved by implementing a persistent polling loop within a dedicated goroutine. This loop continuously queries your data sources—such as database connection pools, message broker depths, or internal APIs—and updates the registered Prometheus metrics in memory.

When designing your collection loop, ensure the polling frequency is more frequent than the Prometheus scrape interval. For example, using a five-second interval for collectMetrics ensures that when Prometheus performs its default fifteen-second scrape, the exporter serves fresh, high-resolution snapshots of your application state. This pattern prevents stale data from being reported in your time-series database.

The following pattern illustrates how to structure this background worker:

• Goroutine Execution: Start the collectMetrics() function in the main() block using the go keyword to ensure it runs concurrently with the HTTP server.
• Metric Updates: Inside the infinite loop, replace simulated values (e.g., rand.Intn(50)) with actual read operations from your underlying system.
• Registration: Use prometheus.MustRegister during initialization to ensure all metrics are registered before the application begins processing requests.

For the HTTP layer, your main() function must handle both metrics exposure and liveness monitoring:

• /metrics: Use the promhttp.Handler() to serialize and serve your registered metrics in the format required by Prometheus.
• /healthz: Register a simple handler function that returns http.StatusOK. This provides a lightweight endpoint for Kubernetes liveness probes to verify the process is running without triggering unnecessary metric processing.

After wiring these handlers and listening on port :8080, verify your implementation locally before deployment. Run the application using go run . and execute curl http://localhost:8080/metrics | grep worker_. If correctly configured, the terminal will display the registered # HELP and # TYPE metadata alongside the current numeric values for your worker_ prefixed metrics.

Multi-stage builds divide the image build process into distinct phases. The first stage compiles the application; the second stage assembles only the artifacts required to run it. After the build completes, Docker discards the builder stage entirely, so compilation tooling, module caches, and environment variables from the build never appear in the final image.

The builder stage starts from golang:1.21-alpine, which contains the Go toolchain and Alpine utilities needed to resolve and download dependencies. CGO_ENABLED=0 forces the Go compiler to produce a statically linked binary with no dependency on libc or other shared libraries. Copying only go.mod and go.sum before the source code leverages Docker's layer cache: dependency downloads are only re-executed when those files change.

The runtime stage starts from gcr.io/distroless/static:nonroot. Distroless images contain no shell, no package manager, and no standard command-line utilities—only the application binary and its direct dependencies. The nonroot variant runs as a non-root user by default, and because the exporter binary is statically linked, copying it alone is sufficient.

FROM golang:1.21-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /exporter .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /exporter /exporter
EXPOSE 8080
ENTRYPOINT ["/exporter"]

This approach yields several concrete benefits:

• A smaller image, which reduces pull time and storage cost.
• No Go toolchain or build cache in production, shrinking the attack surface.
• No shell or package manager, eliminating common vectors for exploitation.
• A non-root default user, satisfying common cluster security policies without extra configuration.

Build and push the image, replacing <registry> with your own registry address:

docker build -t <registry>/my-exporter:v1.0.0 .
docker push <registry>/my-exporter:v1.0.0

Manual execution is acceptable for local testing, but CI/CD automation is generally a better pattern. A pipeline produces reproducible builds, applies credential management centrally, triggers pushes on commits, and can run image scans before a tag becomes available to production clusters.

To integrate your custom exporter into a Kubernetes environment, you must bridge the gap between the ephemeral nature of Pods and the static discovery requirements of Prometheus. This is achieved through a pair of foundational manifests: the Deployment and the Service.

The Deployment manages the Pod lifecycle, ensuring the exporter remains running. Because exporters are typically lightweight processes, you should define conservative resources limits (e.g., 50m CPU and 32Mi memory requests) to prevent resource contention. Crucially, the livenessProbe must point to the /healthz endpoint rather than /metrics to ensure the health check remains decoupled from data collection logic. The following metadata pattern is recommended:

• Labels: Use app.kubernetes.io/name to ensure standard discovery across the cluster.
• Named Ports: Define the container port with a name (e.g., metrics). This allows the Service to reference the port by name rather than hardcoding a numeric value, which facilitates configuration management.

The Service provides a stable network identity for your Pods. By selecting the Pods via the shared label, the Service maps the metrics port to the exposed container port. Once the Service is active, you must configure Prometheus to discover it.

For clusters utilizing the Prometheus Operator or the kube-prometheus-stack, you must deploy a ServiceMonitor custom resource. The operator must be running in the cluster before this resource is created. The ServiceMonitor requires a release label that matches the label selector defined in your Prometheus resource; for a standard Helm installation, this is typically set to kube-prometheus-stack. By matching these labels, the operator automatically updates the Prometheus scrape configuration to include your new target.

Once Prometheus is successfully scraping these metrics, they become available for complex alerting or, more importantly, as the foundation for the HorizontalPodAutoscaler to trigger scaling events based on real-world application performance data.