CalibForge: Stop Filtering Tasks, Start Tuning Them Against Solvers

Paper: 2608.06352 Authors: Fanzhe Meng, Guoxin Chen, Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao, Ruihua Song, Ji-Rong Wen, Kai Jia Categories: cs.LG, cs.CL

The Gap

Everyone building agentic training data right now has converged on roughly the same recipe: take a repo or a container image, have an LLM author a task plus a programmatic verifier, build the environment, and check that the whole thing executes end to end. SWE-smith, SWE-Gym, R2E-Gym, and the various Terminal-Bench-style synthesis pipelines all live in this family. The quality gate is executability plus verifiability: the Dockerfile builds, the test suite runs, the grader returns a clean pass/fail.

That gate answers exactly one question — is this task well-formed? It says nothing about whether the task is useful to train on. A verified task can be trivially solved by any model that can type ls, or it can be unsolvable by anything short of a human expert with a week. Both are near-worthless as supervision: the first yields no gradient signal, the second yields no successful rollouts at all in an RL setting. The field’s usual patch is post-hoc filtering — run the model, keep tasks whose pass rate falls in some band, throw the rest away. That works, but it burns most of what you generated, and it treats difficulty as something you discover by luck rather than something you can build.

CalibForge’s claim is that difficulty isn’t a property of a task at all. It’s a relation between a task and a solver. So instead of measuring and discarding, measure and rewrite.

  PROBLEM
  [agent training needs tasks that are solvable AND
   appropriately hard -- not just well-formed]
      |
      v
  [existing pipelines stop at: does it build? does it verify?]
      |
      +--> blind spot: "hard" is a relation, not an attribute
      |    so a verified task tells you nothing about its
      |    training value
      |
      v
  ASSUMPTION
  [ value(task) = f(task, solver) ]
  [ therefore: probe solvers, then edit the task ]
      |
      v
  METHOD ..... CalibForge
      |
      +-- multi-solver calibration : chase pool disagreement
      +-- contrastive calibration  : force strong-pass / weak-fail
      |
      +-- shared anchor: some solver must have actually passed it
      |
      v
  EVIDENCE
  [ 5,431 calibrated tasks                                    ]
  [ ablation: calibrated > author+validate only                ]
  [ ablation: calibrated > ordinary single-solver feedback     ]
  [ 32.58% / 47.57% on Terminal-Bench 2.0                      ]
  [ max deltas vs base: +24.71 TB2.0, +27.68 SWE-bench Pro,    ]
  [                     +30.04 Doc2Repo                        ]
      |
      v
  CONCLUSION
  [ solver-relative learnability is a *constructible* target,
    and it transfers off-distribution ]

The Increment

One sentence: Before, a synthesized agent task was validated once for executability and then accepted or filtered on pass rate; after, the task is a mutable artifact that gets repeatedly probed by a heterogeneous solver pool and edited until its pass/fail signature matches a targeted difficulty relation.

Core Mechanism

The front half is conventional. Start from a seed environment — a repo, a doc set, a container — and have an authoring model produce the triple that any executable task needs: a natural-language instruction, a programmatic verifier, and an environment spec. Build it, run it, confirm the verifier discriminates. Anything that fails here is dropped, same as everywhere else.

The new part is the loop that follows. The candidate task is handed to a pool of solvers spanning a capability range — different base models, different scaffolds, different budgets — and each produces verified rollouts. What comes back is not a scalar difficulty but a pass/fail signature across the pool. The signature is read as a diagnosis. All-pass means the task is under-specified or trivial. All-fail means it is either genuinely too hard or, more often, broken in a way executability checks didn’t catch — an ambiguous instruction, a verifier that demands an exact string no reasonable agent would produce. Anything in between is potentially in-band.

Two calibration objectives sit on top of this signature. Multi-solver calibration simply targets disagreement: keep editing until the pool splits. It’s cheap to specify and gives you tasks that sit at the frontier of *some model in the pool. Contrastive calibration is sharper: you nominate a strong solver and a weak solver and demand the specific relation *strong passes, weak fails. That pins the task to a chosen difficulty band rather than wherever the pool happens to disagree. When the signature doesn’t match the target, a revising model edits the task — tightening or loosening the instruction, adding a required intermediate step, adjusting what the verifier accepts — and the whole thing goes back through the executable check and another round of rollouts. Both strategies share one hard constraint that the paper calls anchoring in demonstrated solvability: a task is never kept on the theory that it’s solvable. Someone in the pool has to have actually solved it.

   [ seed: repo / doc set / container image ]
                    |
                    v
        +--------------------------+
        | authoring model          |
        | instruction + verifier   |
        | + environment spec       |
        +--------------------------+
                    |
                    v
        +--------------------------+   fail
        | executable validation    |--------> drop
        +--------------------------+
                    | pass (it builds, it verifies)
                    v
  ==============  CALIBRATION LOOP  ==============
                    |
                    v
     +--------------------------------------+
     | solver pool rollouts                 |
     |   S_strong    S_mid    S_weak  ...   |
     +--------------------------------------+
                    |
                    v
          read the pass/fail signature
                    |
        +-----------+-----------+--------------+
        |           |           |              |
     all pass    all fail   disagreement    strong pass
        |           |        in pool        + weak fail
        v           v           |              |
  [ too easy ] [ too hard   ]   +------+-------+
        |      [ or broken   ]          |
        |           |                   v
        +-----+-----+              ACCEPT ..> corpus
              |                          (5,431 tasks)
              v
     +-------------------------+
     | revising model          |
     | add a crux / disambig.  |
     | / retune the verifier   |
     +-------------------------+
              |
              +----> back to executable validation
                     (bounded rounds, then give up)

  anchor rule: never keep a task unless at least one
  solver in the pool has demonstrably passed it

Think of it as a climbing gym route-setter, and the mapping is tight enough to carry the whole method. The authoring model is the setter bolting holds onto a blank wall. Executable validation is checking that the holds are actually screwed in and the top anchor exists — necessary, and completely silent on whether the route is any good. The solver pool is the gym’s regulars: a couple of V8 climbers, some V5s, a few beginners. You don’t grade a route by staring at it; you put bodies on it and watch. Everyone flashes it means the route is boring and you add a crux. Nobody gets off the ground means either it’s a V12 or you’ve set something that’s not physically climbable by human anatomy — and the route-setter’s instinct, exactly like this paper’s, is to suspect the second before the first. The interesting route is the one where the V8 sends and the V5 falls at the third move: that’s contrastive calibration, a designated strong-pass/weak-fail relation. Multi-solver calibration is the lazier version — set for *any split in the gym. And the anchoring rule is the setter’s non-negotiable: a route doesn’t get a grade on the board until someone has topped it, because a route nobody has climbed might be a great problem or might just be broken, and you cannot tell from the wall.

Key Concepts

• Solver-relative learnable zone: There is no such thing as a “hard task” in the abstract, only a task that is hard *for a given solver. “Print the current directory” is trivial for GPT-class agents and impossible for a bash-only script that doesn’t know what pwd means. The learnable zone is the set of tasks a particular solver fails often enough to have something to learn but succeeds at often enough to generate a positive example. Concretely, if you’re doing RL with verifiable rewards, a task at 0% pass rate produces zero reward signal forever and a task at 100% produces zero advantage — the useful mass is in the middle, and where the middle is moves as your model improves. CalibForge’s move is to make that zone a design target you build toward rather than a filter you apply afterward.

• Adversarial solver calibration: “Adversarial” here doesn’t mean gradient attacks or perturbations. It means the task generator is optimizing *against observed solver behavior: it looks at what the solvers did, finds where the task failed to separate them, and edits the task to create that separation. The adversary is the task, not a model. The closest familiar analogue is psychometric item calibration — a test writer pilots a question on real students, sees that 95% got it right, and rewrites it to discriminate better. Same loop, LLMs on both ends.

• Demonstrated vs. assumed solvability: Most pipelines certify that a task is solvable *in principle — a reference solution exists, or the verifier passes when you run the ground-truth patch. CalibForge insists on something stricter: an actual solver in the pool completed the task from the instruction alone, under the same conditions the trainee will face. This distinction does real work. A task can have a perfect reference solution and still be unsolvable from its instruction because the instruction omits a crucial detail, and no amount of executable validation will surface that. Requiring a demonstrated pass makes instruction-quality failures indistinguishable from difficulty failures — which is fine, because in both cases you want to rewrite.

Framework Shift

Before (mainstream):                 After (CalibForge):

  generate task                        generate task
       |                                    |
       v                                    v
  does it build? --no--> drop          does it build? --no--> drop
       | yes                               | yes
       v                                    v
  does verifier work? --no--> drop     probe with solver pool
       | yes                               |
       v                            +------+------+---------+
     KEEP                           |      |      |         |
       |                          all    all    split /   target
       v                          pass   fail   contrast  relation
  (optional) measure pass          |      |         |         |
  rate, filter out-of-band         v      v         v         v
       |                        [ diagnose: too easy /   KEEP
       v                          too hard / broken ]
     train                            |
                                      v
  task = immutable            +----------------+
  difficulty = discovered     | rewrite task   |
  rejects = wasted           +----------------+
                                      |
                                      +--> re-probe (bounded)
                                                 |
                                                 v
                                               train

                                     task = mutable artifact
                                     difficulty = designed
                                     rejects = raw material

One sentence: from validating tasks against a specification to calibrating tasks against a population of solvers, the core shift is that difficulty stops being a discovered property to filter on and becomes an editable target to optimize toward.

Expert Assessment

Problem choice: Real gap, and well-timed. Anyone who has actually run RLVR on synthesized agent tasks has hit this wall — you generate 50k verified tasks, discover 80% are trivial and 15% are impossible, and train on the remaining scraps. The field’s trajectory is clear: 2023 was “can we generate verifiable tasks at all”, 2024–25 was “can we generate them at scale”, and this paper is squarely in the “can we generate them at the *right difficulty” phase. That’s the correct next question. The reframing of difficulty as a relation rather than an attribute is not deep — it’s obvious once stated — but naming it and building a pipeline around it is a genuine contribution, and the contrastive strong-pass/weak-fail formulation is a cleaner objective than the pass-rate-band heuristic most people use.

Method maturity: Clever in framing, mostly engineering in execution. The individual pieces all exist: LLM-as-reviser is Evol-Instruct, pass-rate targeting is standard curriculum filtering, multi-model disagreement as a difficulty proxy shows up in active learning and in data-selection work. The increment is wiring them into a closed loop with a demonstrated-solvability anchor. What worries me is the cost question the abstract sidesteps entirely. Every calibration round costs a full set of agent rollouts across the pool — that is *expensive, and 5,431 tasks is a modest corpus, which smells like a compute ceiling rather than a design choice. The obvious simpler alternative is brute over-generation plus pass-rate filtering at matched compute: generate 10x the candidates, run one cheap probe each, keep the in-band ones, discard the rest. Revision only wins if editing a near-miss is cheaper than generating a fresh candidate that happens to land in-band. I would want that comparison as a headline experiment, and from the abstract it doesn’t appear to be one. The ablation against “ordinary single-solver feedback” is the right control but a weaker one — it holds the loop fixed and varies the signal, which shows the pool helps without showing the loop was worth building.

Experimental integrity: The ablations are the honest part — comparing against authoring-and-validation-only and against single-solver feedback is exactly what a skeptic would ask for, and reporting both calibration strategies separately is good practice. The headline deltas are where I’d apply pressure. “+24.71 points over the corresponding base model” on Terminal-Bench 2.0 implies a base around 8%, and base models evaluated in an agentic harness they were never trained for post absurdly low numbers for reasons that have little to do with capability — tool-call formatting, not knowing when to stop, failing to read the environment. A large chunk of any 25-point gain is the model learning the harness. That’s a real and useful effect, but it is not evidence that *calibration did the work; only the ablation deltas speak to that, and the abstract doesn’t give their magnitude. The 47.57% on TB2.0 is a respectable number for a trained open model without being frontier. Two further flags: Doc2Repo is not a benchmark I recognize as an established third-party evaluation, and a +30-point gain on a benchmark plausibly aligned with the authors’ own seed distribution is much weaker evidence than the SWE-bench Pro number. And there’s an unaddressed circularity — tasks calibrated against a specific solver pool encode that pool’s idiosyncratic weaknesses, so transfer claims lean entirely on the off-distribution results holding up. SWE-bench Pro’s +27.68 is the load-bearing number in the whole paper.

Writing quality: The abstract is precise about the framing and evasive about the mechanism. “Revise candidate tasks” is doing enormous unexplained work — what edit operations does the reviser have? Is it free-form rewriting, or a constrained set of moves like adding a subgoal or perturbing the verifier? How many rounds before giving up, and what fraction of candidates ever converge? That convergence rate is arguably the single most informative statistic about whether this approach is practical, and it’s absent. If I could have one section rewritten, it’d be the revision operator with a yield-and-cost table: candidates in, rounds spent, tasks out, GPU-hours per accepted task. That would turn a plausible-sounding pipeline into something a reader can decide to adopt. Secondarily, the paper needs to be far more careful separating harness-learning gains from capability gains in how it presents the base-model deltas.

Verdict: weak accept — the reframing is correct and the ablations are the right ones, but the cost accounting is missing and the headline numbers lean on base-model comparisons that overstate what calibration itself contributes.

Takeaways

Things worth stealing:

• Read the pass/fail signature as a diagnosis, not a score. The genuinely reusable trick is that all-fail is more often a *broken task than a hard one. If you’re generating verifiable tasks and throwing away everything at 0% pass, you’re discarding your best bug reports about your own generator. Route those back into revision instead.

• Contrastive difficulty targeting beats pass-rate bands. “Strong model passes, weak model fails” is a cheaper and more stable specification than “pass rate between 20% and 80%”, because it needs two rollouts instead of a reliable rate estimate, and it stays meaningful as your models change — you just swap which model plays the strong role. Directly transferable to math, reasoning, or any RLVR dataset construction.

• Demanding a demonstrated pass, not a reference solution. Cheap and underused. If no solver can complete the task from the instruction alone, the instruction is broken, regardless of how clean your ground-truth patch is. This catches a class of underspecification that no amount of executable validation will.

• A weak model is a useful instrument, not just a bad solver. Keeping a deliberately weak solver in the pool as a triviality detector is a nice inversion — you’re using its failures as a difficulty floor.

What not to take on faith: the assumption that revising a task is cheaper than generating a new one. Measure that in your own pipeline before committing to a calibration loop; if your generation cost is low and your rollout cost is high, plain over-generate-and-filter may still dominate.