Not the First Error, but the One That Killed It: TrajDebug and Error-Lifecycle Tracing

Paper: 2608.06346 Authors: Yunjia Qi, Zehua Yin, Xintong Shi, Hao Peng, Songyuanyi Lu, Yixian Liu, Richeng Xuan, Yuhong Liu, Zhichao Hu, Xiaozhi Wang Categories: cs.AI

The Gap

Everyone building agents has the same experience: the agent runs 80 steps, fails, and you’re left scrolling through 150k tokens of tool calls trying to figure out where it went wrong. The research response to this has been “critical error detection” — automatically find the step responsible for the failure. The dominant recipe is LLM-as-judge over the trajectory: dump the whole thing into a strong model, ask it to point at the bad step. Variants add step-by-step scoring, binary-search-style localization (as in some agent-debugging and “who’s to blame” multi-agent failure attribution work), or process reward models trained on step labels.

Two things break this recipe, and the paper names both cleanly.

First, evidence dispersion. Whether step 37 was a mistake often depends on a constraint buried in the system prompt at step 0, an observation returned at step 12, and a decision the agent made at step 20. A judge reading a truncated or flattened trajectory doesn’t have those three things in view simultaneously. Long-context models help less than you’d hope, because the problem isn’t capacity, it’s attention over scattered, low-salience spans.

Second — and this is the sharper point — earliest is not critical. A failed trajectory typically contains several local errors. The agent calls the wrong API, gets an error back, retries correctly. That’s a real error at step 12, and it’s completely irrelevant to why the run failed. Prior work conflates “first thing that looks wrong” with “cause of failure,” which systematically mislocates blame toward early, self-healed slips.

TrajDebug’s bet: criticality is not a property of a step in isolation. It’s a property of an error’s trajectory through time — did it get resolved, and did it reach the end?

[Problem] long failed trajectory, N steps, many local errors
      |
      v
[Prior approach] LLM-as-judge over whole/flattened trace
      .- evidence for step i scattered across s_0 .. s_i
      .- returns EARLIEST error, not CAUSAL one
      .- self-recovered errors get blamed
      |
      v
[Assumption] every error has a LIFECYCLE:
      onset --> (resolved | persists) --> (terminal impact | none)
      critical := earliest error that is unresolved AND reaches failure
      |
      v
[Method] TrajDebug
      + multi-granularity history compression  (fix dispersion)
      + evidence-based error identification    (fix hallucinated verdicts)
      + lifecycle tracing / attribution        (fix earliest != critical)
      |
      v
[Evidence] TrajErrBench: 486 human-annotated failed traces
      from Tau2Bench (tool use) + SWE-Bench Pro (coding)
      .- best overall vs existing baselines
      .- diagnoses fed back --> downstream agent success up
      |
      v
[Conclusion] detection alone is underdetermined;
             attribution needs a survival check

The Increment

One sentence: Before, agent failure localization asked “which step looks wrong?”; after, it asks “which wrongness survived to kill the run?” — turning a step-classification problem into an error-lifecycle tracking problem, with a benchmark that actually distinguishes the two.

Core Mechanism

The pipeline has three stages that map onto the three failures of the prior recipe.

Stage 1: multi-granularity history compression. Instead of feeding the judge either the full trajectory (too long, attention dilutes) or a summary (evidence destroyed), TrajDebug builds a layered context per step under evaluation. A coarse layer carries the task specification and stable global constraints. A middle layer carries compressed summaries of earlier phases — what was attempted, what state the environment reached, what was learned. A fine layer keeps a verbatim window around the step being judged: the exact call, the exact observation. The judge always sees the global rules and the local detail at full fidelity, and pays compression cost only on the middle distance.

Stage 2: evidence-based error identification. Each step gets judged, but the verdict must come with citations — which instruction it violates, which observation contradicts it, which prior commitment it breaks. This is a discipline device more than an interpretability feature: requiring a pointer suppresses the failure mode where a judge, primed by knowing the trajectory failed, declares steps wrong on vibes. The output is a candidate set of local errors, each with a location and a stated reason.

Stage 3: lifecycle tracing and critical attribution. Now the actual novelty. For each candidate error, the framework looks *forward: was this error resolved later — did the agent retry, correct the value, roll back the edit, recover the state? And independently: does its consequence persist into the terminal state that the evaluator marked as failure? Errors that were resolved are pruned regardless of how early or how ugly they were. Among the survivors with terminal impact, the earliest is the critical step. Detection and attribution become two separable passes, which is why the framework can be right about “there was an error at step 12” and still correctly refuse to blame it.

raw trajectory:  s_1 s_2 s_3 ............... s_N   (100k+ tokens)
                                |
                                v
+--------------------------------------------------+
| STAGE 1: multi-granularity history compression   |
|   global : task spec + hard constraints (verbatim)|
|   mid    : phase digests (what tried / state)    |
|   local  : window around s_i (verbatim)          |
+--------------------------------------------------+
                                |
                                v
+--------------------------------------------------+
| STAGE 2: evidence-based error identification     |
|   for s_i:  verdict + citation                   |
|             .- violated instruction?             |
|             .- contradicted observation?          |
|             .- broken prior commitment?          |
+--------------------------------------------------+
                                |
                    candidates  \{ e_12, e_19, e_37, e_52 \}
                                |
                                v
+--------------------------------------------------+
| STAGE 3: error-lifecycle tracing                 |
|   e_k:  resolved downstream?   y/n               |
|         reaches terminal state? y/n              |
|                                                  |
|   e_12  resolved @ s_15 ........ prune           |
|   e_19  resolved @ s_22 ........ prune           |
|   e_37  never resolved + terminal .. KEEP  *     |
|   e_52  downstream of e_37 ...... not earliest    |
+--------------------------------------------------+
                                |
                                v
                  CRITICAL STEP = s_37

The metaphor: an autopsy with the full medical chart.

Think of a failed trajectory as a patient who died, and the debugger as a pathologist who must write one line on the death certificate: cause of death.

The naive judge is a pathologist handed a 400-page chart and told “find the first abnormal reading.” They flip to page 30, see an elevated white cell count, and write “infection.” But the chart shows the infection was treated on page 45 and cleared by page 60. It was a real abnormality and it did not kill the patient.

TrajDebug is the pathologist who works properly. Compression is the chart structure: the admission notes and known allergies stay verbatim at the front (global layer), the intervening months are condensed into ward summaries (mid layer), and the hours around the reading under examination are read line by line (local layer). No competent clinician reads all 400 pages uniformly; they read the intake, the summaries, and the relevant window.

Evidence citation is “point at the lab value.” A diagnosis that can’t name the test result behind it doesn’t go in the report. This is what stops the pathologist from reasoning backwards — “the patient died, so everything looks pathological.”

Lifecycle tracing is the treatment record. For each abnormality: was it treated, and did the treatment take? The elevated white count — treated, resolved, struck from consideration. The medication interaction on page 210 — never addressed, and its downstream effects appear in every subsequent chart entry up to the end. That’s what goes on the certificate. And the organ failure on page 380 is a *consequence, not the cause, so you trace upstream to the earliest surviving lesion.

The load-bearing part of the analogy: a symptom list is not a cause of death. You need onset, treatment, and outcome for each finding. That’s exactly the shift from step classification to lifecycle tracing.

Key Concepts

• Critical error, as distinct from first error: Imagine you’re following a recipe and you grab salt instead of sugar, notice immediately, and put it back. Error made, error fixed, cake fine. Later you set the oven to 250°C instead of 180°C and never notice. The cake burns. If someone asks “what went wrong?”, the honest answer is the oven, not the salt — even though the salt happened first. “Critical error” means the earliest mistake whose consequence was never undone and that shows up in the final bad outcome. The definition is implicitly counterfactual: fix this step and re-run, and the task would plausibly have succeeded. Fixing the salt changes nothing.

• Resolution status (the survival check): Agents are self-correcting systems. They get error messages, they retry, they revert files, they re-query. So a snapshot judgment (“this step is wrong”) is missing a time dimension. Resolution status asks, for each error, whether the world was put back on track afterward. Concretely: agent runs sed on the wrong file at step 12, sees the diff, runs git checkout on it at step 14. The repository state at step 15 is identical to step 11. Whatever killed this run, it wasn’t that. Pruning by resolution status is what removes the systematic early-bias of prior detectors.

• Multi-granularity history compression: The obvious ways to fit a long trajectory into a judge both fail. Truncation drops the constraint you needed. Uniform summarization destroys the token-level detail that distinguishes a correct tool call from a wrong one — “user_id=8891” vs “user_id=8819” survives no summary. The fix is to notice that different distances need different resolutions: the task spec must be exact (it’s the yardstick), the middle history only needs to convey state and intent (it’s context), and the step under examination must be exact (it’s the object of judgment). It’s the same principle as a map with an inset: continent-scale for orientation, street-scale where you’re actually standing.

Framework Shift

Before (mainstream approach):          After (this paper):

  [ full / truncated trajectory ]        [ layered context per step ]
              |                            global  : verbatim spec
              v                            mid     : phase digests
       [ LLM-as-judge ]                    local   : verbatim window
              |                                      |
              v                                      v
     "first suspicious step"              [ judge + evidence citation ]
              |                                      |
              v                                      v
        report step 12                     candidates \{12, 19, 37, 52\}
              |                                      |
              v                                      v
   (step 12 was self-fixed;               [ lifecycle: resolved? terminal? ]
    real cause was step 37)                 12 fixed  x
                                            19 fixed  x
    detection == attribution                 37 alive + terminal  *
                                            52 downstream of 37
                                                     |
                                                     v
                                            critical step = 37

                                        detection =/= attribution
                                        (two separate passes)

From “classify each step as correct or incorrect” to “track each error from onset to outcome,” the core shift is treating an error as an object with a lifespan rather than a label on a timestep — which makes self-recovery a first-class signal instead of noise.

Expert Assessment

Problem choice: Real gap, and well-timed. Anyone who has run a horizon-100 agent knows that failed traces are full of errors the agent shrugged off, and that “first error” metrics reward the wrong thing. The field has been drifting toward this — multi-agent failure attribution work, process reward models, agent-as-judge — but mostly kept the step-classification framing. Naming self-recovery as the confound, and building the evaluation to expose it, is the kind of reframing that’s obvious only in retrospect. It sits squarely on the path from “agents fail” to “agents that repair themselves,” since you can’t do targeted self-repair without correct attribution.

Method maturity: Honestly, the insight is clever and the implementation is conventional. Each stage is a prompted LLM pass with structured output; nothing here requires training. That’s a strength for adoption and a weakness for depth — this is a well-designed pipeline, not a new mechanism. Two things I’d want checked. First, the whole thing is LLM judgments stacked on LLM judgments: if stage 2 hallucinates an error, stage 3 will dutifully reason about whether that fiction was resolved. Errors compound multiplicatively across stages and the abstract gives no stage-wise ablation. Second, a much simpler baseline is lurking: for environments with resettable state (Tau2Bench very much included), you can *re-run from step i with the error corrected and measure whether the task succeeds. That’s a near-ground-truth criticality signal, expensive but decisive, and it would either validate or embarrass the whole pipeline. Also: cost. Per-step judging plus per-error forward tracing is O(N) to O(N·k) LLM calls on a 100-step trajectory. If baselines are one-shot judges, “best overall performance” may be partly a compute-budget artifact.

Experimental integrity: The benchmark is the paper’s most valuable artifact and also where the risk concentrates. 486 manually annotated failed trajectories from two sources is respectable but small, and “which error is critical” is a genuinely hard labeling task — I’d want inter-annotator agreement reported prominently, plus the protocol for resolving disagreement. If agreement on the critical step is, say, 70%, then the headline gap between methods needs to be read against that ceiling. Two domains (tool use, code) is thin for a claim about “long-horizon agents” generally; nothing here covers web navigation, GUI, or multi-agent traces where attribution is hardest. The downstream application study — feeding diagnoses back to improve success rate — is the most persuasive evidence in the abstract, because it’s an end-to-end utility check that’s harder to game than a localization accuracy number. I’d want to know whether the gain survives when the feedback is deliberately made less precise, which would tell us how much of the lift comes from *correct attribution versus merely pointing somewhere plausible.

Writing quality: I’m working from the abstract here, so treat this as calibration on what the abstract signals. “Multi-granularity history compression” and “evidence-based error identification” are doing a lot of unearned work as phrases — they name a shape, not a mechanism, and the abstract gives no hint of what defines a granularity boundary or what counts as valid evidence. The section that would elevate the paper most is the benchmark construction: the operational definition of “critical,” the annotator instructions, the agreement statistics, and the hard cases where two errors both plausibly contributed. That definition is the paper’s real contribution, and if it’s relegated to an appendix the paper will be cited for its method and remembered for its dataset.

Verdict: weak accept — a genuinely better problem formulation plus a useful annotated benchmark, held back by a pipeline of stacked LLM judgments that isn’t validated against the counterfactual re-execution baseline the environments already support.

Takeaways

Things worth stealing, whether or not you care about agent debugging:

• Add a survival check to any root-cause analysis. The generalizable move is: enumerate candidate faults, then ask of each one “was this undone before the end?” This applies directly to alert triage (drop alerts that self-healed), flaky test analysis, incident postmortems, and log anomaly detection. Most detectors report symptoms; almost none check whether the symptom persisted. Adding one forward-looking pass over an existing detector is cheap and will visibly cut false root causes.

• Separate detection from attribution as distinct passes. Bundling “is this wrong?” and “did this matter?” into one judgment is why judges over-blame early steps. Two passes with different context windows — backward-looking for detection, forward-looking for attribution — is a pattern that transfers to any temporal diagnosis task.

• Non-uniform context resolution beats truncation or summarization. Verbatim at the yardstick (spec, constraints) and verbatim at the point of judgment, compressed only in between. If you’re building any long-context judge or retrieval-augmented evaluator, this layout is a better default than “summarize everything” or “keep the last k steps.”

• Mandatory evidence citation as a hallucination brake. Requiring the judge to name the instruction or observation it’s contradicting is a cheap prompt-level constraint that suppresses outcome-biased reasoning (“it failed, so this must be wrong”). Useful anywhere you’re using an LLM to evaluate something whose outcome the model already knows.

• A framing to reuse: criticality is counterfactual, so validate it counterfactually. If your environment supports state reset, the strongest label for “did this step matter” is re-running from it with the step fixed. Even a small counterfactually-verified subset makes a human-annotated benchmark far more trustworthy. This is the check I’d run first on TrajErrBench.