Skill-RM: Unifying Reward Models Through Agent-Based Skill Execution

Paper: 2606.03980 Authors: Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu Categories: cs.LG, cs.CL

The Gap

Current reward models for LLM post-training face a coordination problem. They need to evaluate model outputs using wildly different types of evidence: rule-based verifiers for code correctness, ground-truth answers for factual accuracy, procedural checklists for safety, complex rubrics for creative writing. Each evaluation criterion lives in its own silo with its own interface. The field has no unified mechanism to integrate these heterogeneous signals. Existing approaches either hard-code specific verifiers, or train separate reward models for each task, or use general-purpose LLM judges that ignore available structured evidence. The result: reward models that are brittle, opaque, and inefficient at leveraging the rich evaluation resources already available.

Problem: Heterogeneous evaluation criteria
   |
   v
Assumption: Reward computation can be structured as agentic task
   |
   v
Method: Skill-RM (reusable reward-evaluation skills)
   |
   v
Evidence: Outperforms baselines on benchmarks + RL applications
   |
   v
Conclusion: Dynamic orchestration > static evaluation

The Increment

One sentence: Before this paper, reward models were monolithic evaluators bound to fixed criteria; after it, reward models become orchestrators that dynamically select and compose evaluation evidence as reusable skills.

Core Mechanism

Skill-RM reframes reward modeling as executing a structured “Reward-Evaluation Skill” — a reusable program that defines how to gather and aggregate evidence for a specific evaluation task. The skill consists of three layers: (1) a skill definition that specifies what evidence types are relevant (verifiers, references, rubrics), (2) a dynamic evidence selection module that decides which resources to invoke based on the input, and (3) an aggregation component that combines signals into a final reward score.

When evaluating a model output, Skill-RM first analyzes the input and task context to determine which evaluation resources are applicable. For a coding problem, it might invoke a unit test verifier and code quality checker. For a creative writing task, it might retrieve a rubric and reference examples. The skill then executes each selected component, collects their outputs (pass/fail signals, similarity scores, rubric grades), and aggregates them using learned or rule-based weights. Crucially, the skill is modular and reusable — the same skill definition can be applied across different inputs within a task category.

Input + Task Context
   |
   v
[Skill Definition] ---> Evidence Types: \{verifier, reference, rubric\}
   |
   v
[Dynamic Selection] ---> Active: \{verifier, rubric\}
   |                      Inactive: \{reference\}
   v
[Execute Components]
   |
   +---> Verifier: pass/fail
   |
   +---> Rubric: 0.85 score
   |
   v
[Aggregate Signals] ---> Final Reward: 0.92

Think of Skill-RM as a restaurant kitchen manager, not a single chef. The manager (skill definition) knows the full menu of cooking techniques available: grilling, steaming, frying, baking. When an order comes in (input), the manager reads the dish requirements (task context) and decides which techniques to use — a steak needs grilling and a sauce reduction, while a dessert needs baking and plating. Each technique (evidence component) is executed by a specialist (verifier/rubric/reference), and the manager combines their outputs (aggregation) into the final dish (reward score). The key insight: instead of training a single chef to do everything mediocrely, you organize specialists and dynamically decide who contributes to each dish.

Key Concepts

• Reward-Evaluation Skill: Instead of treating reward computation as a monolithic neural network or fixed rubric, Skill-RM defines it as a structured program. A skill specifies: “For this type of evaluation, these evidence sources are relevant, here’s how to query them, and here’s how to combine their signals.” This turns reward modeling from a black-box prediction into a transparent, debuggable workflow. Concrete example: A math problem skill might define three steps — (1) extract ground-truth answer from dataset, (2) run symbolic verifier on model output, (3) compute string similarity as fallback. Each step is explicit, and you can swap components (use a different verifier) without retraining the whole model.

• Dynamic Evidence Orchestration: Not all evidence is relevant for all inputs. A code generation task might have unit tests available, but a creative writing prompt does not. Skill-RM learns to conditionally activate evidence sources based on input features. This is different from static pipelines (always run all checks) or manual routing (human-written if-then rules). The model examines the input, task metadata, and available resources, then decides which components to invoke. Example: For a coding problem tagged as “string manipulation,” the skill might skip heavy type-checking verifiers and focus on output format validation.

• Heterogeneous Signal Aggregation: Evidence comes in incompatible formats — binary pass/fail from verifiers, continuous similarity scores from references, ordinal ratings from rubrics. Skill-RM must translate these into a unified reward scale. The aggregation layer learns to weight and combine signals appropriately for each task. For high-stakes tasks (medical Q&A), verifier signals might dominate; for open-ended tasks (creative writing), rubric scores might matter more. The weights are task-specific but learned from data, not manually tuned.

Framework Shift

Before (mainstream approach):        After (this paper):

[Input] --> [Monolithic RM]         [Input] + [Task Context]
              |                                |
              v                                v
         Single Score              [Skill Definition]
                                           |
                                           v
                                  [Evidence Selection]
                                    /      |      \
                                   v       v       v
                              Verifier Reference Rubric
                                   \      |      /
                                    v     v     v
                                  [Aggregation]
                                       |
                                       v
                                  Unified Reward

Static evaluation pipeline       Dynamic orchestration graph

From a monolithic reward predictor to a compositional orchestrator, the core shift is treating reward modeling as an agent-based task where evidence selection and aggregation are explicit, reusable operations.

Expert Assessment

Problem choice: This is a real gap. The proliferation of specialized verifiers, rubrics, and reference datasets has created an integration nightmare. Practitioners either ignore available structured signals (wasteful) or write bespoke pipelines for each task (not scalable). The paper correctly identifies that the bottleneck is not lack of evaluation resources, but lack of a unified interface to use them. The problem sits at a mature inflection point — enough heterogeneous resources exist to make unification valuable.

Method maturity: The approach is conceptually clean but mechanically straightforward. The “skill” abstraction is essentially a structured program with learned routing. The novelty lies in the framing (agentic task execution) rather than technical breakthreck. The dynamic selection module is just attention over evidence types, and the aggregation is weighted averaging. That said, simplicity is a strength here — the method is practical and implementable without exotic components. The risk: it might be over-engineered for tasks where a single strong verifier suffices.

Experimental integrity: Baselines are reasonable (GPT-4-as-judge, task-specific RMs), and the benchmark coverage is broad (RewardBench, downstream RL tasks). Numbers show consistent but not dramatic improvements (~2-5% on most metrics). The ablation studies are thin — we don’t see clear evidence that dynamic selection outperforms static always-use-all strategies. The RL experiments (best-of-N, policy training) are convincing but small-scale. Red flag: no analysis of failure modes or cases where orchestration overhead exceeds benefits.

Writing quality: The abstract and introduction are sharp. The method section buries key details in dense notation — the skill definition formalism (Section 3.2) needs a worked example before the math. The related work undersells how this connects to tool-use and planning literature, which have solved similar orchestration problems. Figure 2 (skill execution flow) does heavy lifting; Figure 3 (benchmark results) is hard to parse due to cramped layout. Rewriting the method section with code-like pseudocode instead of mathematical notation would elevate clarity.

Verdict: Weak accept — solid engineering contribution with clear practical value, but lacks the conceptual depth or empirical rigor for a strong accept. The unification framing is useful, but the method is incremental rather than paradigm-shifting.

Takeaways

Steal the abstraction, not the implementation. The idea of treating evaluation as skill execution — defining what evidence is relevant, routing dynamically, aggregating transparently — transfers to any domain with heterogeneous quality signals. If you’re building content moderation systems (multiple classifiers + user reports + regex filters), medical diagnosis tools (lab tests + imaging + patient history), or financial risk models (credit scores + transaction patterns + macro indicators), this framework applies. The specific takeaway: make your evidence selection logic explicit and debuggable. Write it as a program (if input has X, invoke Y) rather than hoping an end-to-end model figures it out. The transparency alone justifies the abstraction cost.