Introducing On-Demand CPU & Memory Profiling with Flamegraphs for Workers and Durable Objects

Traditional observability pipelines rely on logs and aggregate metrics, which are useful for detecting that a service is “slow” or “out‑of‑memory” but do not reveal where the work is happening inside the code. Logs capture discrete events and require developers to instrument every point of interest; they cannot show how much CPU time a particular function consumes. Aggregate metrics (e.g., CPU % or memory GB per instance) summarize resource usage at the process level, masking the distribution of work across call stacks. Consequently, engineers often spend time guessing which functions are responsible for spikes, leading to trial‑and‑error optimizations.

On‑demand CPU and memory profiling addresses these gaps by sampling the runtime at a high frequency (typically one‑millisecond intervals) and producing a flamegraph that maps each rectangle to a concrete function. The width of the rectangle directly represents the proportion of CPU cycles or memory allocations attributable to that function, giving precise, function‑level visibility without additional instrumentation.

• CPU profiling example: In a high‑traffic Worker that handles R2 bindings, a flamegraph highlighted genericR2JsonReplacer consuming over 5 % of CPU time. The function recursively walked the JSON tree while JSON.stringify also traversed it, resulting in duplicated work. Refactoring eliminated the recursion and yielded a 2.7× speed‑up.
• Memory profiling example: A Worker exceeded its 128 MB limit (P999 ≈ 133 MB). A heap profile showed that disabled Prometheus instrumentation still allocated ~66 % of memory. Removing the stray code reduced the P999 to 118 MB, providing headroom below the limit.
• Operational workflow: Engineers can request a profile via the Cloudflare CLI (`cf workers versions profile …`) or the dashboard’s Observability tab. The profiler acquires the isolate lock only for start/stop operations, allowing normal request processing to continue during the sampling window.

Because profiling runs against a live isolate, the data reflects real‑world traffic patterns, request distribution across data centers, and the actual state of stateful Durable Objects. This production‑centric view is essential for compliance frameworks such as SOC 2 or ISO 27001, where evidence of performance under realistic load is required.

In practice, teams should:

1. Select a version of the service that receives sufficient traffic.
2. Capture short (e.g., 5 s) CPU or memory profiles during typical load.
3. Inspect the flamegraph to identify the widest functions.
4. Correlate findings with source code (enable source maps for TypeScript) and apply targeted optimizations.

To collect an on‑demand CPU or memory profile of a Cloudflare Worker you must first decide whether to use the cf CLI or the Cloudflare Dashboard. Both interfaces require the same underlying data – a deployed version that receives sufficient traffic – but they differ in workflow.

CLI workflow

1. Install the cf package (e.g., npm i -g @cloudflare/cli) and authenticate with cf login.
2. Identify the Worker you want to profile. You can use the script name or the numeric worker-id shown in the dashboard.
3. Choose the version to profile. The CLI example uses latest, but any published version identifier is valid.
4. Determine the profiling duration in milliseconds. A typical value is 5000 (5 seconds), but longer runs capture more samples.
5. Run the profiling command, redirecting the binary .pprof output to a file for later analysis:

   cf workers versions profile latest \
     --worker-id "$WORKER_ID_OR_NAME" \
     --duration-ms 5000 \
     --profile-type cpu > worker-cpu.pprof
   

6. Open the resulting .pprof file with a compatible viewer (e.g., pprof or Chrome DevTools) to inspect the flamegraph.

Dashboard workflow

1. Log in to the Cloudflare Dashboard and navigate Build → Compute → Workers & Pages.
2. Select the target Worker from the list.
3. Open the Observability tab and choose “Flamegraph” from the dropdown.
4. Use the version selector to pick the deployed version you wish to profile.
5. Enter the desired duration (e.g., 5000 ms) and choose the profile type (CPU or memory).
6. Ensure the selected version is receiving “plenty of traffic”; low‑traffic versions may not produce a usable profile.
7. Click Capture Profile. After the interval elapses, the dashboard renders an interactive flamegraph and offers a download of the raw profile.

Key considerations

• Traffic volume: Profiling requires enough request samples; a version with minimal traffic may yield sparse data.
• Version selection: Profile the version that reflects the code you intend to optimize; older or test versions can mislead analysis.
• Source‑map support for TypeScript: Enable source maps in your build pipeline; otherwise the flamegraph will display obfuscated function names, making it difficult to map performance hotspots back to the original TypeScript source.
• Profile type: CPU profiles reveal time‑consuming functions, while memory profiles expose allocation hotspots.

In a flamegraph each horizontal rectangle corresponds to a single function call that occurred while the profiler sampled the runtime. The rectangle’s width is proportional to the cumulative amount of the selected resource—CPU time for a CPU profile or allocated bytes for a memory profile—that the function consumed across all samples. Because the graph stacks calls vertically, a wide rectangle that appears near the top indicates a hot path that dominates the overall cost, while narrow rectangles represent relatively inexpensive work.

The interactive view lets you hover over a rectangle to see the exact function name, the number of samples, and the percentage of total time or memory. Clicking a rectangle isolates its call stack, allowing you to focus on the sub‑graph of callers and callees that contribute to that function’s cost.

Using the table view

The table view presents the same data in a sortable list of functions. Typical columns include:

• Function – the demangled name (source maps are required for TypeScript).
• Samples – raw count of profiler ticks that hit the function.
• % of total – proportion of the resource represented.
• Self‑time – time spent in the function itself, excluding children.

Sorting by Samples or % of total quickly surfaces the “widest” functions without scanning the visual graph.

Practical workflow for optimization

1. Capture a profile of a production Worker or Durable Object for a duration that yields sufficient samples (e.g., cf workers versions profile … --duration-ms 5000 --profile-type cpu).
2. Inspect the flamegraph; note any rectangles that dominate the width.
3. Switch to the table view and sort by Samples. Identify functions that contribute > 5 % of the total (as in the genericR2JsonReplacer example where it accounted for over 5 % of CPU time).
4. Examine the call stack of each wide function. Look for patterns such as recursive calls, duplicated work, or unnecessary external library invocations (e.g., repeated metrics() calls or a partially‑disabled Prometheus collector that accounted for ~66 % of allocations).
5. Apply targeted code changes—deduplicate calls, eliminate redundant recursion, or remove unused instrumentation—and re‑profile to verify the width reduction.

By iterating this process, engineers can prioritize the functions that consume the most resources, achieve measurable reductions in CPU cycles or memory allocations, and avoid over‑optimizing low‑impact code paths.

Profiling in production provides the most reliable view of where CPU cycles and heap allocations are spent. Cloudflare’s on‑demand CPU and memory flamegraphs let engineers capture a live isolate, then examine the widest functions in the visual or table view. The process isolates the runtime lock only while the profiler starts and stops, so normal request handling continues uninterrupted.

Optimizing genericR2JsonReplacer

The R2 binding worker handled high traffic, and a 50‑second CPU profile showed genericR2JsonReplacer consuming over 5 % of CPU time. The function was invoked by JSON.stringify for every node, but it also recursively walked the same JSON tree, causing each nested value to be processed multiple times. By removing the internal traversal and letting the replacer act only on the current node, the recursive work was eliminated, yielding a 2.7× speed‑up.

Eliminating duplicate metric calls

A second hotspot was a repeated call to registry.metrics(). The original code resembled:

function ship() {
  if (!registry.metrics().length) { return; }
  // …
  return Response(registry.metrics());
}

Because metrics() performs a heavy aggregation, calling it twice added roughly 1 % of total CPU time. Caching the result in a local variable reduced the overhead to near zero.

Resolving an OOM caused by partially‑enabled Prometheus code

A worker exceeded the 128 MB memory limit (P999 ≈ 133 MB). Heap profiling revealed that the Prometheus instrumentation, which should have been disabled, still executed and accounted for about two‑thirds of allocations. Removing the stray code path freed memory across all percentiles, as shown below.

Key takeaways for engineers

• Use on‑demand flamegraphs to locate the widest functions before refactoring.
• Inspect the table view for hidden recursive patterns or duplicated calls.
• Validate that optional instrumentation (e.g., Prometheus) is fully disabled in production builds.
• Measure before‑and‑after metrics (CPU time, memory percentiles) to confirm the impact of changes.

When a profiling request is issued, the Workers runtime must observe the live isolate that is already handling traffic. To avoid halting request processing, the runtime uses an isolate lock strategy that limits the time the lock is held to short lifecycle operations only. The sequence is:

• Acquire the isolate and its exclusive lock.
• Instantiate a V8 CPUProfiler and start sampling at a one‑millisecond interval.
• Immediately release the lock so normal JavaScript execution can continue while samples are collected.
• After the requested duration elapses, reacquire the lock, stop the profiler, and release the lock again.
• Serialize the profile data outside the critical section and return it to the caller.

This approach guarantees that the isolate remains responsive; holding the lock for the full profiling window would block all incoming requests and render the profile meaningless.

For Durable Objects (DOs), the same mechanism applies but with additional routing considerations because a DO is a stateful actor identified by a name. The runtime first resolves the live primary instance of the named object, determines the data center where that instance currently resides, and then applies the isolate‑lock workflow to that specific isolate. This enables engineers to target a precise DO instance rather than an arbitrary replica.

The profiling request is an HTTP POST to the Cloudflare API. A minimal example looks like:

curl 'https://api.cloudflare.com/client/v4/accounts/<account_id>/workers/workers/<worker_name>/versions/latest/profile' \
  --data-raw '{"duration_ms":5000,"profile_type":"cpu"}'

The URL encodes both the script name and the version to be profiled. The runtime uses the version identifier to locate the most recent deployment that has active isolates in the target data center. If the isolate is not currently loaded (e.g., low traffic), the request will return an error because there is no live execution context to sample.

In summary, profiling on the edge proceeds as follows:

• Identify the requested version and data center from the API payload.
• Locate the corresponding isolate (or DO primary actor).
• Apply the isolate‑lock strategy with V8 sampling at 1 ms.
• Collect samples without pausing request handling, then serialize the flamegraph data.