From Flat Logs to Execution Trees: Debugging Modern AI Agents

Agent traces are typically written as a sequence of span_started and span_ended events because append-only data is simple to produce. For example, a research agent might emit span_started, span_started, span_ended, and so on. Developers rarely want to read that flat sequence. They want causal structure: the research_agent as the root, with children such as search_web, query_database, call_finance_api (including retries), and summarize_results.

The event stream is optimized for writing; the execution tree is optimized for understanding. Building that tree reliably is harder than sorting timestamps because:

• Events may arrive out of order; a buffered exporter can deliver an end event before its start.
• Siblings may run concurrently, so adjacency in the log implies neither parentage nor sequence.
• Spans may be incomplete; a crashed process or dropped exporter batch may never deliver an end event.
• Retries can fail while the parent operation still succeeds, so child status must not automatically overwrite parent status.

To reconstruct the tree, every event needs stable traceId and spanId values. Start events establish parentage; end events establish outcome and duration. An assembler can buffer end events until their start arrives, report duplicate starts/ends, and flag spans left open instead of guessing a completion time. For live streams, pending events require a size limit and expiry policy.

The assembler should produce a forest, not necessarily a single tree. Multiple roots are valid if trace stitching is incomplete, and orphaned spans must be displayed separately with diagnostics. Cycles in parent links need detection before recursive rendering. Retries and fallbacks should be modeled as child spans with individual outcomes. The parent can legitimately succeed after all retries fail if a fallback succeeds; only the operation itself defines contract success. Open spans are evidence, not clutter, and should be rendered as incomplete rather than silently closed at the trace’s last timestamp.

Establishing a reliable execution tree requires more than sequential event logging. Because distributed systems often generate events that arrive out of order, overlap, or fail prematurely, timestamps alone are insufficient to reconstruct causality. Two events adjacent in a stream may be siblings, unrelated concurrent processes, or operations from entirely different traces. To resolve this, engineers must adopt a formal contract that enforces stable identity and explicit parentage.

The minimum event contract requires two discrete event types: span_started and span_ended. These events utilize consistent identifiers—traceId, spanId, and parentSpanId—to form a directed acyclic graph, ensuring that hierarchical relationships persist regardless of arrival sequence.

The schema relies on the following structure:

• span_started: Captures traceId, spanId, parentSpanId (null for root spans), name, kind (run, model, tool, retrieval, decision, or fallback), and timestampMs.
• span_ended: Captures traceId, spanId, timestampMs, status (ok, error, or cancelled), errorCategory, and arbitrary metadata.

By decoupling the start and end of an operation, this contract accounts for real-world failure modes, such as crashed processes that never emit an end event. An assembler can handle these via buffering, where span_ended events are held until the corresponding span_started is processed. This approach avoids inventing durations or status outcomes for incomplete operations. Any span left without an end event should be explicitly represented as "open," allowing developers to distinguish between successful operations, errors, and instrumentation gaps. Because trace reconstruction is a prerequisite for debugging, using explicit parentSpanId links rather than inferred temporal proximity is the only way to ensure that nested work—such as a tool call initiated by a decision agent—is accurately rendered within the execution tree.

An append-only event stream is optimized for writing, not for reconstructing execution trees. A buffered exporter can deliver an end event before its matching start; a crashed process may never deliver the end at all. The SpanAssembler therefore buffers end events in a pendingEnds map until the corresponding start event arrives. Assembly state is explicit: each AssembledSpan carries a status of open, ok, error, or cancelled, along with identity, parentage, timestamps, and metadata. The assembler never assumes that a span without an end is successful.

Four diagnostics make this behavior observable:

• duplicate_start — a start event arrives for a span that is already tracked.
• duplicate_end — an end event arrives for a span that is already closed or already buffered.
• end_without_start — an end remains buffered when the stream is finalized.
• span_left_open — a start was never matched with an end.

When an end event arrives first, it is buffered without altering any span. When the start arrives, the assembler applies the buffered end, setting endedAtMs to Math.max(startedAtMs, event.timestampMs) to avoid a negative duration. It does not invent missing timestamps or promote an open span to ok. Incomplete spans remain open, with the reason available via diagnostics.

For an unbounded live stream, the pending map must have a size limit and an expiration policy. A malicious or malformed sequence of end events for unknown spans would otherwise grow memory without bound. Bound the pending map by count and by time, evicting stale entries with a duplicate_end or end_without_start diagnostic as appropriate.

A trace emitted by an instrumented system is, at rest, an append-only event stream. Start events establish parentage; end events establish outcome and duration. Timestamps alone cannot reconstruct that structure: two adjacent events can be siblings, concurrent work, or fragments of different traces. A renderer first assembles start and end events into spans, then builds an execution tree from the parent links those spans declare.

Even after assembly, a valid trace normally has one root. Buffered exporters, crashed processes, and dropped end events can produce spans whose parent never appears, or multiple top-level operations within one trace ID. A renderer must therefore tolerate multiple roots and orphaned spans without crashing. The forest structure keeps those cases observable rather than exceptional:

type SpanNode = AssembledSpan & { children: SpanNode[] };

type TraceForest = {
  roots: SpanNode[];
  orphans: SpanNode[];
  duplicateIds: string[];
};

function buildForest(spans: AssembledSpan[]): TraceForest {
  const nodes = new Map<string, SpanNode>();
  const duplicateIds: string[] = [];

  for (const span of spans) {
    if (nodes.has(span.spanId)) {
      duplicateIds.push(span.spanId);
      continue;
    }
    nodes.set(span.spanId, { ...span, children: [] });
  }

  const roots: SpanNode[] = [];
  const orphans: SpanNode[] = [];

  for (const node of nodes.values()) {
    if (node.parentSpanId === null) {
      roots.push(node);
      continue;
    }
    const parent = nodes.get(node.parentSpanId);
    if (!parent) {
      orphans.push(node);
      continue;
    }
    parent.children.push(node);
  }

  const sortChildren = (node: SpanNode): void => {
    node.children.sort((a, b) =>
      a.startedAtMs - b.startedAtMs || a.spanId.localeCompare(b.spanId)
    );
    node.children.forEach(sortChildren);
  };

  roots.sort((a, b) => a.startedAtMs - b.startedAtMs);
  roots.forEach(sortChildren);

  return { roots, orphans, duplicateIds };
}

Children are ordered by start time and then span ID to produce a stable display without implying causality between siblings. Before applying that sort recursively, the renderer must validate parent links for cycles. A malformed trace in which A is the parent of B and B is the parent of A will otherwise cause infinite recursion. A depth-first search with visiting and visited sets detects cycles in linear time; any node reached while it is already in the visiting set can be moved to the orphan list with a diagnostic such as cycle_detected.

Orphans should not be merged into the root list or silently discarded. They belong in a separate “unattached spans” section with diagnostics — for example parent_missing or cycle_detected — so an operator can distinguish an instrumentation gap from genuinely missing work. Hiding those spans makes a partial trace look complete.

Use the forest structure to keep rendering robust:

• Render every root as a separate tree, never assuming a single root.
• Render orphans under an explicit unattached section with per-span diagnostics.
• Deduplicate by span ID, preserving the first occurrence and reporting the duplicate.
• Run cycle detection before any recursive render pass.

Visualizing execution traces requires distinguishing between logical hierarchy and chronological duration. While sorting sibling spans by their start time provides a stable UI display, engineers must recognize that this order does not imply causality. Because children often execute concurrently, their intervals frequently overlap, rendering linear sequence assumptions invalid.

Effective observability tools must present two distinct views to accurately convey the state of a system:

• Tree View: Optimized for inspecting parentage, retries, fallbacks, and internal handoffs. This view clarifies the why behind operations.
• Timeline View: Optimized for visualizing overlaps, wait states, time to first output, and critical latency. This view clarifies the when behind operations.

A frequent error in performance analysis is the summation of individual span durations to determine total latency. Because parent spans encapsulate the duration of their children, and parallel child spans overlap in wall-clock time, arithmetic summation often leads to gross overestimation. For example, a root span lasting 2 seconds containing two parallel 1-second tool spans results in a total wall-clock time of 2 seconds, not the 4 seconds produced by summing all spans.

To avoid misrepresenting performance, categorize metrics precisely:

• Trace Wall Time: The total duration from root start to root end.
• Span Duration: The interval calculated as endedAtMs - startedAtMs for a specific operation.
• Self Time: The duration of a span excluding the union of its child intervals.
• Critical Path: The chain of dependencies dictating completion time.

Engineers must be aware that calculating a true critical path requires explicit dependency semantics, not merely parentage. A trace schema must encode decision rules—such as whether a tool execution was a required dependency, an optional fallback, or an alternative attempt—before a UI can confidently label a critical path. Without these rules, the system cannot programmatically distinguish between blocking dependencies and background concurrent work.

Robust distributed tracing requires shifting from linear log aggregation to structured execution trees. By modeling operations as hierarchical nodes, engineers can move beyond simple status propagation to nuanced observability of complex workflows.

Retries, Fallbacks, and Status Propagation

Retries and fallbacks must be represented as distinct child spans, each maintaining its own identity and outcome. Because a parent operation frequently succeeds despite initial child failures, automated status propagation—where a parent assumes the worst status of its children—is often misleading. Instead, the parent status should reflect whether the overall contract was fulfilled. Implement quality gates that monitor for specific indicators of sub-optimal performance, such as:

• Operations that succeeded only after exceeding a pre-defined attempt budget.
• Successful parent executions that relied on stale fallback data (e.g., cache hits with high age attributes).
• Total retries exceeding historical norms for a specific span kind.

Handling Partial Traces

Incomplete spans serve as critical diagnostic evidence, not mere interface clutter. When an exporter or process terminates prematurely, span states such as open or cancelled must be preserved. Avoid the anti-pattern of silently closing spans at the trace’s final timestamp, as this creates artificial durations and misrepresents the system's actual failure state. Instead, render incomplete spans distinctly by documenting the termination reason where available:

• Completion missing: Signal that the event stream was truncated before an end event arrived.
• Client disconnected: Indicate that the consumer terminated the request, interrupting the lifecycle.
• Adapter error: Flag cases where a downstream adapter terminated before returning a callback.

If a user interface provides a visual estimation of a range for incomplete spans, it must explicitly label that duration as an estimate rather than an observed metric.

Invariant-Based Comparison

Comparing execution trees using exact snapshots is fragile due to inherent variance in asynchronous environments. To perform reliable regressions, compare durable invariants rather than raw duration or timestamp data. Focus comparisons on the following properties:

• Structural Integrity: Validate required and forbidden span kinds within specific branches.
• Relational Mapping: Ensure essential parent-child relationships remain consistent across versions.
• Operational Efficiency: Compare counts of attempts, retries, and fallback invocations.
• Terminal Status: Verify that the outcome (e.g., ok vs. error) of the root node and primary branches remains stable under identical inputs.