SWE-Prime: Training on 10% of the Trajectories Beats Training on All of Them

Paper: 2608.27449 Authors: Dewu Zheng, Ruizhe Ye, Yanlin Wang, Yang Ye, Hongyu Zhang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jianxing Yu, Zibin Zheng Categories: cs.AI, cs.CL, cs.SE

The Gap

The recipe for improving coding agents has been: collect large-scale agent trajectory datasets, keep the successful ones, and run supervised fine-tuning on them. Success is the label.

The paper’s objection is that success is a result label and it is being used as a process label. A trajectory can resolve the issue and still contain ineffective, redundant, or risky steps — dead ends that were abandoned, repeated exploration of the same file, a command that worked here but is not a pattern you would want imitated. Fine-tuning on it teaches the model to imitate those steps, because SFT has no way to know which parts of a trajectory were instrumental.

This is a specific and worsening problem. As trajectory datasets scale, the fraction of steps in any given trajectory that actually contributed to the resolution probably falls — long successful trajectories accumulate more wandering. So the standard recipe gets worse as the data grows, and it gets worse precisely in the direction the field has been scaling.

   THE STANDARD RECIPE

   collect large-scale agent trajectories
        |
        v
   keep the SUCCESSFUL ones  <-- success is the label
        |
        v
   supervised fine-tuning on them
        |
        v
   [THE OBJECTION] success is a RESULT label being used
                   as a PROCESS label
        |
        v
   a successful trajectory can still contain:
     ineffective steps  |  redundant exploration
     risky commands     |  abandoned dead ends
        |
        v
   SFT imitates ALL of it -- it cannot tell which steps
   were instrumental to the resolution
        |
        v
   [GAP] and it WORSENS with scale: longer successful
         trajectories accumulate more wandering

The Increment

One sentence: Before this paper, the success of a trajectory was the criterion for training on it; after it, two-stage filtering at trajectory and segment granularity means training on 10% of the resolved data outperforms training on all of it.

Core Mechanism

The method is a multi-granularity, two-stage selection procedure, and the granularities are the point — the two stages answer different questions.

Stage one: trajectory-level screening, on three criteria — process quality, result quality, and data representativeness. So a trajectory must not merely succeed; it must succeed well, and it must add something the dataset does not already have. The representativeness criterion is the subtle one: it prevents the filtered set from collapsing onto the easiest task types, which is what a pure quality filter would do.

Stage two: segment-level selection. Consecutive steps are grouped into semantic segments — units that correspond to an intent rather than to a fixed window of steps — and each segment is assessed on its contribution to the final solution, its learnability, and its potential risks. Contribution separates instrumental work from wandering; learnability separates segments that teach something transferable from those that are too task-specific; risk flags segments you would rather not have imitated at all.

The mechanism that makes segment-level selection work is a detail worth pausing on: during SFT, all segments remain in the sequence to preserve context, while only selected segments contribute to the loss. This is the right design and it is non-obvious. Dropping unselected segments would break the causal chain — the model would see a step without the context that produced it, and the remaining steps might not make sense. Keeping them in the input while masking them from the loss means the model is trained to predict only the good steps, while still conditioned on the full history that led to them.

The results are stated in the form that makes the argument: training on the 10% trajectory subset selected by SWE-Prime outperforms training on the full resolved dataset, with relative gains of up to 12.2% on SWE-Bench Pro and 24.2% on SWE-Bench Verified.

Read that comparison carefully, because it is stronger than a usual data-efficiency claim. The baseline is not “a smaller random subset” — it is the full resolved dataset. So the extra 90% of trajectories were not merely unhelpful; they were actively harmful on net. That is the quantitative form of the paper’s premise: the redundant and risky steps in successful trajectories are not neutral filler that washes out, they impose a cost.

   SWE-PRIME: TWO STAGES

   [ all successful trajectories ]
                |
                v
   STAGE 1: TRAJECTORY-LEVEL SCREENING
     process quality      (did it succeed WELL?)
     result quality
     data representativeness  (does it ADD anything?)
                |
                v
   STAGE 2: SEGMENT-LEVEL SELECTION
     group consecutive steps into SEMANTIC SEGMENTS
     score each on:
       contribution to the final solution
       learnability
       potential risks
                |
                v
   TRAINING DETAIL
     ALL segments stay IN THE SEQUENCE  (preserve context)
     ONLY selected segments contribute to the LOSS
        -> model predicts the good steps,
           conditioned on the full history
                |
                v
   RESULT: 10% trajectory subset  >  FULL resolved dataset
     SWE-Bench Pro        up to  12.2% relative
     SWE-Bench Verified   up to  24.2% relative
        -> the other 90% was NET HARMFUL, not merely unhelpful

Think of it as learning surgery by watching recordings of operations, where you can only tell afterwards which ones went well. Keeping every recording of every successful operation teaches the trainee a great deal that has nothing to do with competence: the time the surgeon explored the wrong area, the instrument passed twice because it was misidentified the first time, the improvisation that happened to work on that patient. A competent surgeon’s recording and a lucky one’s look identical if your only filter is “the patient survived”. What you want is to keep the recordings, so the trainee sees each step in context, but mark only the segments worth imitating — the incision done properly, not the fumbling that preceded finding it.

Key Concepts

• Result label versus process label: success tells you the trajectory ended correctly, not that its steps were worth imitating. Treating them as interchangeable is the assumption the paper removes, and it is why a success-filtered dataset can be worse than a smaller filtered one.
• Two granularities for two questions: trajectory level asks “is this a good and useful example at all?”; segment level asks “which parts of it should be learned?”. Collapsing them into one score loses the ability to keep a valuable trajectory while discarding its bad stretches.
• Context preservation with loss masking: keeping all segments in the input while only selected segments contribute to the loss. It is what allows segment-level filtering without breaking the causal chain that makes the remaining steps interpretable.
• Net harm, not just redundancy: the 10% subset beating the full dataset means the discarded trajectories cost more than they contributed. Distinguishing “unhelpful” from “harmful” changes what you do about data volume.

Framework Shift

Before (success as the label):
  collect trajectories -> keep successes -> SFT on all of them
  -> success correlates with quality but does not measure it
  -> more data monotonically assumed to help

After (quality as the label, at two granularities):
  trajectory level: process, result, representativeness
  segment level: contribution, learnability, risk
  train on selected segments with context preserved
  -> 10% subset BEATS the full resolved dataset
  -> the removed data was net harmful

From using task success as the training criterion, to selecting both which trajectories and which parts of them to learn from, the core shift is that a successful demonstration is not the same as good supervision, and the difference is measurable.

Expert Assessment

Problem choice: Excellent, and it names an assumption the field has been running on. “Keep the successes” sounds like a quality filter and is in fact a result filter, and the paper’s point that the gap widens with scale — longer successful trajectories accumulating more wandering — explains why the standard recipe gets worse as datasets grow, which is the opposite of the usual scaling assumption.

Method maturity: The two-stage design is well motivated, and the segment-level stage is the substantive contribution: grouping into semantic rather than fixed-size units respects the structure of a trajectory, and scoring on contribution, learnability and risk addresses three distinct ways a segment can be bad. The context-preservation-with-loss-masking detail is the kind of implementation choice that makes the difference between a method that works and one that does not, and it is stated plainly.

Experimental integrity: The baseline choice is what makes the result convincing. Comparing against the full resolved dataset rather than against a smaller random subset means the finding is about quality, not about sample efficiency, and reporting that the 10% subset wins demonstrates the discarded data was net harmful. Two benchmarks with different difficulty profiles — Pro and Verified — showing 12.2% and 24.2% means the effect is not a single-benchmark artefact. The main open question is how much of the gain comes from each stage and each criterion; the paper’s contribution would be easier to adopt if the ablation isolated the representativeness filter, which is the least obvious of the criteria.

Writing quality: The premise is stated as a crisp contrast — task success versus high-quality supervision — and the results are reported in the comparison that carries the argument. What would strengthen it is one worked example: a trajectory that passed the success filter and failed trajectory-level screening, with the offending steps identified, so a reader can calibrate their own data.

Verdict: strong accept — it identifies a labelling assumption that gets worse with scale, fixes it at two granularities, and demonstrates the fix with a comparison that shows the old data was harmful rather than merely unhelpful.

Takeaways

• Treat task success as a result label, not a quality label. Filter on process, then filter again at the segment level before training.
• Preserve context while masking the loss. Removing filtered steps from the input breaks the causal chain; keep them visible and exclude them from the gradient.
• Compare against the full dataset, not a smaller random subset. Only that comparison reveals whether the extra data was neutral or harmful.
• Expect the problem to get worse as you scale trajectories. Longer successful runs contain proportionally more wandering, so a success filter degrades exactly where you are investing.