From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills

Paper: 2605.23899 Authors: Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai Categories: cs.AI

The Gap

Language agents are learning to reuse skills—procedural knowledge extracted from past experience. The field has produced many extraction methods, but no one has systematically studied whether these model-generated skills actually help. Prior work focuses on extraction algorithms without measuring end-to-end utility: Does the extracted skill improve the agent’s performance on new tasks? When does it help versus hurt? What makes a skill transferable?

The gap is empirical and conceptual. We have extractors but no utility-grounded evaluation spanning the full lifecycle: experience generation → skill extraction → skill consumption. Without this, we’re building in the dark.

Problem: Skills extracted, utility unknown
   |
   v
Assumption: Utility depends on full lifecycle, not just extraction
   |
   v
Method: Evaluate across 5 domains, multiple extractors/consumers
   |
   v
Evidence: Skills help on average but cause negative transfer;
          extractor/consumer strength decouples from utility
   |
   v
Conclusion: Lifecycle analysis reveals utility predictors;
            meta-skill guides extraction toward useful features

The Increment

One sentence: Before this paper, we had skill extraction methods but no systematic understanding of when extracted skills actually help agents; after, we have a lifecycle framework showing skills cause non-trivial negative transfer and a meta-skill that reduces it.

Core Mechanism

The paper builds a three-stage evaluation pipeline. First, experience generation: collect trajectories from agents solving tasks in a domain. Second, skill extraction: apply different extraction methods (prompting models to distill procedures from trajectories). Third, skill consumption: give extracted skills to target agents and measure task performance change.

The framework tests five domains (ALFWorld, WebShop, SciWorld, GAIA, InterCode-SQL), multiple extractor models (GPT-4, Claude, Llama), and multiple consumer models. For each combination, they measure utility: does the skill improve success rate? The key innovation is treating utility as the ground truth, not extraction quality in isolation.

Experience Pool          Extractor           Consumer Agent
(trajectories)          (LLM)               (LLM + skill)
                                            
[Task 1: steps]  --->   Extract   --->    [New task]
[Task 2: steps]  --->   patterns  --->    + skill library
[Task 3: steps]  --->   into      --->    = performance
      ...                skills            
                                            
                        ^                   ^
                        |                   |
                   What makes a         Does it
                   good skill?          actually help?

Think of it like a recipe refinement system. You watch chefs (experience generation), write down their techniques as recipes (skill extraction), then give those recipes to new cooks (skill consumption). The question isn’t whether the recipe sounds good—it’s whether cooks using it produce better dishes. This paper measures the dishes, not the recipe prose.

The structural metaphor is load-bearing: experience = watching chefs, extraction = writing recipes, consumption = cooking with recipes, utility = dish quality. Each lifecycle stage maps cleanly. The insight: we’ve been judging recipes by how they read, not by whether they make people better cooks.

Key Concepts

• Negative transfer: When a skill hurts performance instead of helping. Imagine giving a recipe for French pastry to someone making Chinese dumplings—the technique doesn’t just fail to help, it actively confuses. The paper finds this happens frequently with model-generated skills: 20-40% of skills decrease success rate. This isn’t noise; it’s systematic. Skills extracted from one context can encode assumptions or procedures that mislead in another. The key insight: skill utility isn’t monotonic. More skills ≠ better performance.

• Extractor-consumer decoupling: A model can be great at extracting skills but bad at using them, or vice versa. Think of a food critic who writes excellent recipes but can’t cook, versus a chef who cooks brilliantly but writes confusing instructions. The paper shows GPT-4 extracts high-utility skills but performs poorly when consuming skills from others. Claude shows the opposite pattern in some domains. This decoupling means you can’t predict skill utility from model scale or baseline task performance—you need lifecycle evaluation.

• Meta-skill: A skill about making skills. Instead of extracting procedures directly from experience, the meta-skill is a prompt that guides extraction toward features correlated with utility (specificity, actionability, domain-grounding). It’s like teaching someone to write recipes by showing them what makes recipes useful: concrete measurements, clear steps, common failure modes. The paper derives this meta-skill from lifecycle analysis, then shows it improves extraction quality across domains and reduces negative transfer by 30-50%.

Framework Shift

Before (mainstream approach):        After (this paper):

Experience --> [Extractor] --> Skill   Experience --> [Extractor] --> Skill
                                                           |              |
                                                           v              v
                                                      [Consumer] --> Utility
                                                           ^
                                                           |
                                                    Measure & analyze
                                                    
Focus: Extraction quality              Focus: End-to-end utility
Eval: Does skill look good?            Eval: Does agent perform better?
Assumption: Good extraction            Finding: Extraction quality ≠
            = useful skill                      consumption utility

From extraction-centric to utility-grounded, the core shift is measuring what matters: whether skills actually help agents solve new tasks, not whether they seem well-formed in isolation.

Expert Assessment

Problem choice: Real gap. The field has been building extraction methods without systematic utility evaluation—classic “build first, measure later” trap. The problem sits at a critical juncture: as agents scale, skill reuse becomes essential, but we’re flying blind on what makes skills transferable. The timing is right.

Method maturity: Solid empirical work, not a clever trick. The lifecycle framework is straightforward but necessary—someone had to do the tedious work of testing all combinations. The meta-skill derivation is where it gets interesting: they analyze what features correlate with utility, then encode those features as extraction guidance. This is more principled than it sounds—they’re closing the loop from measurement to method improvement. No simpler approach was overlooked; this is the right level of complexity for the question.

Experimental integrity: Baselines are fair. Five domains with different characteristics (embodied, web, science, QA, code), multiple models, consistent evaluation protocol. The negative transfer finding is the most important result, and they show it’s robust across domains. One concern: the meta-skill evaluation uses the same domains where features were analyzed. Cross-domain generalization would strengthen the claim, but the consistency across five diverse domains is reassuring. Numbers hold up.

Writing quality: The paper front-loads motivation well but buries key findings in dense tables. Section 4 (lifecycle dissection) is where the insights live, but it reads like a laundry list. Rewriting this section with clearer narrative structure—“here’s what we found about experience composition, here’s what it means for extraction”—would elevate the whole paper. The meta-skill derivation (Section 5) should be more prominent; it’s the actionable contribution.

Verdict: strong accept — First systematic utility-grounded evaluation of model-generated skills, revealing non-trivial negative transfer and providing actionable guidance for extraction.

Takeaways

For practitioners building agent systems: Don’t assume more skills = better performance. Test skills on held-out tasks before deploying. If you’re extracting skills, bias toward specificity and actionability over generality—the paper shows vague skills cause negative transfer.

For researchers: The extractor-consumer decoupling is the sleeper insight. Model scale doesn’t predict skill utility, which means you can’t shortcut evaluation by using the biggest model. You need lifecycle testing. This applies beyond skills to any learned artifact agents reuse (tools, plans, heuristics).

Concrete technique: The meta-skill prompt structure (guide extraction toward domain-specific, actionable, failure-aware procedures) is immediately usable. Append it to your extraction prompts. The paper shows 30-50% reduction in negative transfer across domains.

Transferable framing: Utility-grounded evaluation. Whenever you’re building a component that feeds into a larger system, measure end-to-end impact, not component quality in isolation. This applies to retrieval systems, tool libraries, prompt templates—anything that’s supposed to help but might hurt.