Learning Globally Reusable Skills for Coding Agents

Paper: 2608.06153 Authors: Chen Yang, Jiashuo Tian, Ziqi Wang, Xinyin Liu, Meiru Ye, Junjie Chen Categories: cs.SE, cs.AI

The Gap

Automated skill evolution is the appealing answer to a real constraint. Retraining an LLM agent is expensive; letting it accumulate reusable skills — procedures, heuristics, code snippets it writes down and consults later — gives you continuous improvement at inference-time cost. The agent solves a task, notices what worked, writes it into a skill bank, and is better next time.

The trouble is in how that writing-down happens. Existing approaches treat skill evolution as a sequence of local updates: a task comes in, the agent succeeds or fails, and a skill gets added or revised in response to that task. Each update is locally sensible and globally unaccountable. Two consequences follow, and they are distinct problems that tend to get conflated.

The first is a compatibility problem. Skills in a bank are not independent — they overlap, subsume one another, and are invoked in sequence. Updating one skill in isolation can silently contradict another, so the bank drifts into internal inconsistency that no single update is responsible for. Nothing in a local-update regime is looking at the bank as a whole.

The second is a generalization problem. A skill written in response to one task encodes that task’s incidentals along with its lesson. The result is skills that are really memorized solutions wearing the costume of a general procedure — they fire on the task that produced them and fail on its neighbors. Worse, because each update is validated (if at all) against the task that prompted it, the overfitting is invisible at the moment it is introduced.

Skill evolution as a sequence of LOCAL updates
         |
         +-------------------+-------------------+
         |                                       |
         v                                       v
  skills interact but are            update validated only
  updated in isolation               against its own task
         |                                       |
         v                                       v
  bank drifts into                   skills overfit; fail
  inconsistency                      on neighboring tasks
         |                                       |
         +-------------------+-------------------+
                             |
                             v
         [This paper] GSE: optimize compatibility
         AND generalization globally
                             |
     +-----------------------+-----------------------+
     |                       |                       |
     v                       v                       v
 Skill Relation      cluster-based           replay-driven
 Graph (co-evolve    consolidation           verification
 relationships)      (abstract reusable      (catch behavioral
                     capability)              regressions)
                             |
                             v
         Evidence: best P/R/F1 on 2 SE tasks x
         2 agents; +61.4% F1 on an industrial agent

The Increment

One sentence: Before this paper, an agent’s skill bank grew by appending locally-validated updates and quietly rotted; after it, skill evolution is a global optimization over a structured bank, with regressions caught before they land.

Core Mechanism

GSE (“globalized skill evolution”) attacks the two failures with three components, and the cleanest way to read the design is that each component answers one of the problems above.

For compatibility, GSE maintains a Skill Relation Graph (SRG). Rather than storing skills as a flat list, it explicitly models inter-skill relationships — which skills overlap, which depend on which, which are specializations of others — and co-evolves those relationships as skills change. This is the structural move that makes global consistency checkable at all: once relationships are first-class objects, an update to a skill has a defined blast radius, and the neighbors it might contradict are enumerable rather than hypothetical.

For generalization, GSE does two things. Cluster-based skill consolidation groups related local updates and abstracts a reusable capability out of them. The logic is that a single task-specific update cannot tell you which of its details are essential and which are incidental — but several updates arising from related tasks can, because what survives across the cluster is the general part. Consolidation is thus a way of extracting signal from repetition rather than trusting any one instance. Replay-driven verification then guards the result: before an evolved skill is accepted, it is re-run against previously handled cases to check that it has not broken behavior that used to work. This is regression testing for skills, and it is the direct answer to overfitting being invisible at introduction time — you cannot see it against the originating task, so you look at the other tasks instead.

The evaluation targets two software-engineering tasks that are unusually well chosen for this: bug-revealing test generation and false-positive bug report filtering. Both are tasks where a skill can plausibly generalize (there are real reusable patterns in how bugs are provoked and how spurious reports look) and where overfitting is easy to detect (a test-generation skill that only works on the file it was learned from is obviously useless). Across two state-of-the-art coding agents, OpenHands and mini-SWE-agent, GSE achieves the best precision, recall, and F1. Against existing evolution techniques the reported margins are wide and lopsided: precision up 6.1%–34.1% and recall up 31.8%–180.0% on test generation, and precision up 15.4%–96.4% with recall up 13.1%–19.8% on false-positive filtering. Deployment on an internal industrial agent yields a 61.4% F1 improvement.

                  GSE evolution cycle

   local updates from recent tasks
        |  |  |  |
        v  v  v  v
   +--------------------------+
   | cluster-based            |   group related updates,
   | consolidation            |   abstract the shared part
   +--------------------------+
        |
        | candidate skill
        v
   +--------------------------+
   | Skill Relation Graph     |   place it; find neighbors
   | (overlap / depends-on /  |   it may contradict;
   |  specializes)            |   co-evolve the edges
   +--------------------------+
        |
        v
   +--------------------------+
   | replay-driven            |   re-run past cases:
   | verification             |   did anything regress?
   +--------------------------+
        |
     +--+---------------------+
     |                        |
  regression              clean
     |                        |
     v                        v
  reject / revise      commit to skill bank


   Contrast with local evolution:

   task --> update --> append.        (no graph, no cluster,
                                       no replay)

Here is the structural metaphor. A skill bank evolving through local updates is a codebase where every change is merged after passing only the test the author wrote for it. Each commit is defensible on its own terms. The author saw a problem, wrote a fix, wrote a test that the fix passes, merged. Nobody is wrong at any step. And yet the codebase decays, for exactly two reasons that any engineer will recognize.

First, changes interact. Your fix touches a module that three other modules depend on, and nothing in your review surfaced that, because dependencies were never written down — they lived in whoever happened to remember them. The Skill Relation Graph is the dependency graph made explicit: now a change has a known set of neighbors, and “what else does this touch” is a query rather than a hope.

Second, a fix written against one bug report encodes that report’s specifics. The general lesson and the incidental detail arrive fused, and you cannot separate them from a single instance. What experienced engineers do is wait until the same shape of fix shows up three or four times and then refactor out the common abstraction — because the part that recurs is the part that was real. That is cluster-based consolidation.

And the thing that keeps the whole practice honest is the regression suite: before merging, run everything that used to pass and confirm it still does. A fix that breaks old behavior gets caught by the old tests, not by its own. That is replay-driven verification. GSE’s contribution, read through this metaphor, is unglamorous and correct — it gives skill banks the engineering discipline that codebases needed and got decades ago.

Key Concepts

• Skill Relation Graph: An explicit, co-evolving model of how skills relate to each other. The word doing the work is co-evolving: it is not enough to build a graph once, because the relationships change as skills change — consolidating two skills into one abstraction rewires everything that pointed at either. Concretely, if the bank holds a skill for “generate a test that triggers a null-pointer path” and another for “generate a test that triggers an unchecked-index path,” the graph records that these are siblings specializing a common parent about boundary-condition provocation. When one is revised, the other is a candidate for the same revision, and any abstraction over both must remain consistent with each.

• Cluster-based skill consolidation: Abstracting a reusable capability from a group of related local updates rather than from any one of them. The underlying epistemics are worth stating plainly, because they generalize far beyond this paper: a single success does not distinguish the cause from its accompaniments, but repeated successes under varying conditions do, since the varying parts cancel and the constant part remains. Consolidation is that cancellation operationalized. This is also why it cannot be replaced by simply prompting the model to “write a general skill” — generality asserted at authoring time is a guess, while generality extracted across a cluster is evidence.

• Replay-driven verification: Re-running an evolved skill against previously handled cases to detect behavioral regressions before acceptance. The reason this is necessary and not merely nice: the failure mode being guarded against is specifically invisible to the validation you would naturally perform. An overfitted skill passes its own task by construction. The only place the damage shows up is on tasks the update was not written for, so those tasks have to be kept around and re-run. It is the same argument that justifies keeping old tests in a suite forever.

Framework Shift

Before (local skill evolution):     After (GSE):

  task_i                              batch of local updates
    |                                        |
    v                                        v
  update skill_j                      cluster + consolidate
    |                                        |
    v                                        v
  validate on task_i                  place in Skill Relation Graph
    |                                  (neighbors made explicit,
    v                                   edges co-evolved)
  append to flat bank                        |
                                             v
  bank = list                          replay past cases
  relations = implicit                       |
  validation = local                         v
                                       accept only if no regression

                                       bank = graph
                                       relations = explicit
                                       validation = global

From growing a skill bank by appending locally-validated updates, to optimizing it as a structured whole, the core shift is making inter-skill relationships and past behavior into first-class objects that every update has to answer to.

Expert Assessment

Problem choice: This is a real gap and an increasingly urgent one, though not a conceptually surprising one. Skill libraries have become standard equipment for coding agents, and the failure mode the paper names — skills that overfit to the task that spawned them — is something practitioners hit immediately and have mostly handled with ad-hoc pruning. The genuinely good move is separating compatibility from generalization. These get lumped together as “the skill bank got worse,” and they have different causes and different fixes; naming them apart is most of the contribution’s clarity. The paper sits comfortably in the field’s trajectory rather than redirecting it.

Method maturity: The three components are sensible and each is well matched to its failure mode, but none is a new idea — this is dependency graphs, refactoring-by-repetition, and regression testing, transplanted into skill banks. That is a compliment about fit and a caveat about novelty. The transplant is worth doing and the paper does it competently. What I would push on is whether all three are load-bearing. Replay-driven verification alone plausibly captures a large share of the benefit, since it is the only component that directly detects the headline failure, and the paper’s contribution would be sharper with an ablation that says how much the SRG buys once replay is in place. There is also an unexamined cost: maintaining and co-evolving a relation graph over a growing bank is not free, and the paper’s framing does not engage with how this scales as banks reach thousands of skills.

Experimental integrity: Two agents (OpenHands, mini-SWE-agent) and two task families is a reasonable spread, and the industrial deployment is the most persuasive single data point precisely because it is the least controlled — a 61.4% F1 improvement on an internal agent is the kind of result that is hard to obtain by benchmark-fitting. The reported ranges deserve a careful read, though. A recall improvement spanning 31.8% to 180.0% is not one effect; it is a range wide enough that the low end and the high end are probably different phenomena, and the honest question is what distinguishes them. Very large relative gains on recall also often indicate a weak baseline on the low-recall end rather than a strong method, and the paper should preempt that reading with absolute numbers. Precision and recall both improving is the encouraging part, since the cheap way to move either is to trade against the other.

Writing quality: The abstract is efficient and the component-to-problem mapping is legible, which is more than most systems papers manage. The corner cut is in the results presentation: leading with wide percentage ranges over unstated baselines invites exactly the skepticism above. The section that would most elevate the paper is a proper ablation with absolute numbers, which would convert a set of impressive-looking ranges into a defensible account of which mechanism does what.

Verdict: weak accept — Correct diagnosis and a well-matched set of borrowed remedies, with results that are probably real but presented in a way that makes them harder to trust than they need to be.

Takeaways

Three things a practitioner can steal:

1. Never accept a learned skill on the evidence of the task that produced it. This is the single most portable idea here and it costs almost nothing to adopt. Keep a replay set of previously handled cases and re-run any evolved skill, prompt, or heuristic against it before committing. The failure you are looking for is structurally invisible to the validation you would otherwise do, because an overfitted artifact passes its originating task by construction.

2. Abstract from clusters, not from instances. When you want a general procedure out of an agent’s successes, do not ask the model to write one after a single win — wait for several related wins and extract what they share. A single success cannot separate cause from accompaniment; repetition under varying conditions can, because the incidentals differ and cancel while the essential part stays put. This applies to prompt libraries, playbooks, and internal documentation as much as to agent skills.

3. Write down the relationships in any growing artifact bank. Whether it is skills, prompts, tools, or MCP servers, the moment the collection is large enough that items overlap and depend on each other, implicit relationships become the thing that rots. Making “what does this overlap with, what depends on it” an explicit and maintained structure converts an unbounded consistency problem into a bounded neighborhood query.