Building the Jig Instead of Training the Hands: Test-Time Capability Transfer via Harnesses

Paper: 2608.12307 Authors: Cheng Qian, Wenting Zhao, Liangwei Yang, Heng Wang, Jielin Qiu, Heng Ji, Silvio Savarese, Huan Wang, Shelby Heinecke Categories: cs.LG, cs.AI, cs.CL

The Gap

Everything we call “distillation” assumes the transfer happens in the student’s weights. Classic teacher forcing trains the small model on the big model’s outputs. On-policy distillation samples from the student and has the teacher score or correct those samples. Reverse-KL variants, sequence-level RL against a teacher reward, rationale distillation — different objectives, same commitment: capability lives in parameters, so to move it you must edit parameters. That costs GPUs, requires weight access, and produces a student you have to re-train when the task changes.

There’s a second, mostly separate literature on inference-time structure: chain-of-thought prompting, self-consistency, prompt optimizers like APE and OPRO, program-of-thought, and agent frameworks such as DSPy. But those are almost always self-improvement — one model optimizing its own scaffold — or a human writing the scaffold by hand. The question this paper picks up is the crossing of those two lines: can a strong model author the scaffold that a weak model runs inside, and does that count as capability transfer? Nobody had measured it cleanly, and it matters, because if the answer is yes then part of what a frontier model “knows” is transferable as an artifact rather than as a gradient.

PROBLEM: capability transfer is assumed to require
         weight updates (teacher forcing, on-policy
         distillation, RL-from-teacher)
              |
              v
ASSUMPTION: part of what the strong model has is
            *cognitive structure* -- decomposition,
            control flow, format discipline --
            which can be externalized as an artifact
              |
              v
METHOD: strong "builder" writes an inference harness
        around a frozen weak "target"; refines it over
        rounds against a 5% validation slice; freezes it
              |
              v
EVIDENCE: 4 Theory-of-Mind benchmarks, avg 0.49 -> 0.91
          ablations attribute gains to:
            (a) reasoning offloaded into deterministic code
            (b) benchmark-specific routing
            (c) strict answer-format enforcement
          NOT to longer reasoning or wider sampling
              |
              v
CONCLUSION: test-time harness design is a distinct
            transfer channel, complementary to
            training-time distillation; weakest
            targets gain most

The Increment

One sentence: Before, “strong helps weak” meant the strong model produced training signal; after, it can produce a *program — and the program alone recovers most of the gap, with the weak model’s weights untouched.

Core Mechanism

The setup has three roles. A builder (strong model) never answers test questions. A target (weak model, frozen) answers all of them. Between them sits the harness: an executable artifact — code plus prompt templates plus a parser — that the builder writes. The harness decides how each input is preprocessed, which pipeline it goes down, what the target is asked, what gets computed deterministically instead of asked, and how the target’s raw text becomes a graded answer.

The loop is deliberately cheap. Each benchmark donates 5% of its data as a validation slice. Round one: the builder inspects the task, writes a first harness, and it is run with the target inside it. The builder sees the validation score and, presumably, failure cases; it edits the harness. Repeat for several rounds. Then the harness is frozen and evaluated once on the full test set. No gradients anywhere in this picture — the only thing that “learns” is a text/code artifact, and the only optimizer is the builder’s own reasoning over validation feedback.

What the ablations say the harness actually does is the interesting part, and it is less romantic than the framing suggests. Gains come from three places. First, offloading: for Theory-of-Mind tasks, tracking who-was-in-the-room-when is a state machine, and the builder writes that state machine in Python instead of hoping the target keeps it straight in prose. Second, routing: the harness recognizes item types and dispatches them to specialized pipelines. Third, format enforcement: a lot of apparent incapability was the weak model producing an unparseable answer, and the harness pins the output shape. Notably, the things that did *not help much are the things the prompting literature loves: telling the target to think longer, or sampling it more times.

   [ BUILDER  (strong, no test access) ]
         |
         |  reads task description + 5% val slice
         v
   +--------------------------------------+
   |  HARNESS  (code + prompts, an        |
   |            inspectable artifact)     |
   |                                      |
   |   . input parser / normalizer        |
   |   . ROUTER: item type -> pipeline    |
   |   . deterministic code blocks        |
   |       (state tracking, set logic,    |
   |        belief bookkeeping)           |
   |   . prompt templates for the parts   |
   |       that genuinely need language   |
   |   . output format enforcer           |
   +--------------------------------------+
         |
         |  calls the target as a subroutine
         v
   [ TARGET  (weak, FROZEN weights) ]
         |
         v
     answer  ==>  score on 5% val
         |
         +---------- round k feedback ----------+
         |            (builder edits harness)   |
         |<-------------------------------------+
         |            repeat N rounds
         v
     FREEZE harness  ==>  run once on full test  ==>  0.91

Think of a machine shop. The builder is the senior toolmaker; the target is a junior machinist with unsteady hands. Conventional distillation is apprenticeship: the senior watches the junior cut metal for six months until the junior’s hands themselves get steady. Expensive, slow, and you cannot hand steady hands to the next junior. This paper instead has the senior build a jig: a fixture that clamps the workpiece, with mechanical stops that make the wrong cut physically impossible. The deterministic code blocks are those stops — the junior isn’t asked to eyeball the depth, the jig sets it. The router is the senior deciding which of three jigs to reach for depending on the part. The format enforcer is the go/no-go gauge at the end of the bench: a part that isn’t within spec never leaves. The 5% validation slice is test cuts on scrap stock — the senior makes a few, sees where the jig binds, files it down, tries again. And the punchline of the ablations, in jig terms: the junior didn’t get better at machining. The jig got good enough that machining skill stopped being the bottleneck. Which also tells you the jig’s limits — it fits *this part. Hand it a different part and you’re back to needing a toolmaker.

Key Concepts

• Harness: Not a prompt. A prompt is a sentence you hand a model; a harness is the whole apparatus around the model call — the code that runs before it, the branching that decides which call to make, the loop that makes several calls, and the code that cleans up after. Concretely: instead of asking “Sally left the room, then Anne moved the marble. Where does Sally think the marble is?”, a harness parses the story into a list of events, runs a Python loop maintaining a dict of \{agent: last_observed_location\}, and only asks the model the one sub-question that genuinely needs reading comprehension. The model’s contribution shrinks; the system’s reliability rises.

• Strong-to-weak scaffolding: The transfer channel here isn’t knowledge, it’s *organization. The strong model knows that this class of problem is a state-tracking problem, knows to separate parsing from inference, knows to validate output shape. That know-how is expressible in code. So a frozen 8B model wrapped in a well-designed harness can behave like something much larger — not because it learned anything, but because someone else made the decisions it was bad at making.

• Offloading vs. eliciting: Two ways to get more out of a weak model. *Eliciting means coaxing better reasoning out of it — think step by step, sample twenty times and vote. Offloading means taking the reasoning away from it and giving that subtask to a Python interpreter, which is never inconsistent. The paper’s ablation is that essentially all the gain here is offloading. That’s a real finding, and it’s slightly deflationary: the weak model’s reasoning was never upgraded; its reasoning was routed around.

Framework Shift

Before (mainstream distillation):     After (this paper):

  [ TEACHER ]                           [ BUILDER ]
      |                                     |
      | logits / traces / rewards           | writes + iterates
      v                                     v
  +-----------+                        +-------------+
  | training  |                        |  HARNESS    |
  |   loop    |  GPUs, data, weights   |  (code +    |  a few val rounds
  +-----------+                        |   prompts)  |
      |                                +-------------+
      v                                     |  wraps
  [ STUDENT ]  weights changed              v
      |                                 [ TARGET ]  weights UNCHANGED
      v                                     |
  better student, permanently               v
  transfer stored IN parameters         better system, right now
                                        transfer stored IN an artifact

From weights to artifacts, the core shift is that capability transfer no longer has to be a training problem — it can be a tooling problem, where the strong model’s contribution is a program you can read, version, and audit.

Expert Assessment

(Caveat: I’m working from the abstract’s claims about the paper’s own ablations; I haven’t inspected the harnesses themselves, and that’s exactly where the paper lives or dies.)

Problem choice: Real gap, well-timed. Two literatures — distillation and agent scaffolding — have been running in parallel with almost no contact, and the “who authors the scaffold” axis genuinely hadn’t been isolated. It also lands on a practical question people actually have: I have API access to a frontier model and a small model I must serve; can the former help the latter without a fine-tuning run? Useful framing, and the “weaker targets gain most” result is the kind of scaling-style regularity that makes a paper citable.

Method maturity: Clever framing, thin machinery. The method *is “let a good model iterate on a script against a validation set” — which is well-trodden in prompt/program optimization; the novelty is who’s holding the pen and who’s being helped. That’s fine. What’s less fine is that the paper’s own diagnosis undercuts its headline story. If the gains are deterministic code plus benchmark-specific routing plus format enforcement, then what’s being transferred is closer to a benchmark-specific solver than to “cognitive structure.” A simpler baseline is conspicuously implied and, from the abstract, unaddressed: let the builder write the harness and then swap in a *much stronger target — how much of the 0.91 is the harness rather than any transfer at all? Relatedly, what does a hand-written harness by a competent engineer score? If the answer is 0.90, the builder model is a convenience, not a mechanism.

Experimental integrity: This is where I’d push hardest. Theory-of-Mind benchmarks are the worst possible venue for this claim, because most of them (ToMi-lineage especially) are template-generated. A strong builder with 5% of the data and several refinement rounds is in an excellent position to reverse-engineer the generator, and a router that dispatches on item type is precisely what reverse-engineering looks like. Nothing about that is cheating — the protocol is honest, the harness is frozen before test — but “0.49 to 0.91” then measures *how templated the benchmark is as much as it measures transfer. The paper needs a held-out test the abstract doesn’t mention: build the harness on benchmark A, evaluate on benchmark B. Without cross-benchmark transfer, “transfers cognitive structure” is an overclaim; “recovers benchmark-specific solution structure” is what was shown. Four benchmarks in one narrow domain also makes the generality claim shaky — I’d want at least one domain where the reasoning genuinely can’t be compiled into a state machine.

Writing quality: The abstract is unusually honest — most authors would have buried “gains come primarily from offloading into deterministic code” in an appendix, and putting it up front earns trust. The corner cut is almost certainly the qualitative side: the single most valuable section would be a full harness listing with an annotated diff across refinement rounds, plus a taxonomy of what the builder added and when. Rewrite that and the paper stops being a benchmark-delta report and becomes a study of what “cognitive structure” concretely consists of. Second cut: the near-total absence, judging by the abstract, of a cost accounting. Builder tokens across rounds versus the cost of a LoRA fine-tune is the comparison every practitioner will make.

Verdict: borderline, leaning weak accept — the question is sharp and the honest ablations are worth publishing, but without cross-benchmark generalization the headline number measures benchmark structure at least as much as capability transfer.

Takeaways

• The “offload, don’t elicit” heuristic is the transferable lesson. When a weak model underperforms on a task, first ask which sub-part is deterministic bookkeeping and move it to code. Empirically that beat both longer reasoning and wider sampling here. This generalizes far beyond ToM: entity tracking, unit conversion, date arithmetic, constraint checking, tabular aggregation.
• Measure how much of your “reasoning failure” is a parsing failure. Format enforcement showing up as a *primary gain driver is a mild indictment of a lot of small-model evaluation. Before you fine-tune, pin the output schema and re-measure.
• Use the strong model as a compiler, not an oracle. Cheap pattern to steal: give the frontier model a task spec and a tiny labeled slice, have it emit a *program that calls your cheap model, iterate a few rounds, freeze. You pay frontier prices once, at build time, then serve small forever. The artifact is inspectable and version-controllable, which fine-tuned weights are not.
• Builder reasoning effort scaling monotonically with harness quality is a practical dial: if the harness is mediocre, turn up the builder’s thinking budget before changing anything else.
• The cautionary steal: if your evaluation is template-generated, a build-and-freeze loop over a validation slice will find the templates. Treat any large jump from this recipe as suspect until you’ve tested the harness on a differently-generated dataset.