New Skills or Sharper Primitives? A Probabilistic Perspective on the Emergence of Reasoning in RLVR

Paper: 2602.08281 Authors: Zhilin Wang, Yafu Li, Shunkai Zhang, Zhi Wang, Haoran Zhang, Xiaoye Qu, Yu Cheng Categories: cs.CL, cs.LG

Abstract

This paper investigates whether Reinforcement Learning with Verifiable Rewards (RLVR) gives language models genuine new reasoning abilities or merely activates dormant ones. The authors propose a probabilistic framework with a central hypothesis: the emergence of complex reasoning is driven by the sharpening of atomic step probabilities. Using the Algebrarium framework, models are trained on single-step operations and tested on unseen multi-step tasks. Key findings include: RLVR encourages exploration of previously inaccessible reasoning paths; composite task performance correlates strongly with the joint probability of atomic steps (Pearson correlation 0.69-0.96); and RLVR can sacrifice specific skills to maximize overall reward.

Key Contributions

• Probabilistic framework: A formal model connecting atomic step probabilities to composite reasoning performance
• Sharpening hypothesis: RLVR improves reasoning by increasing the probability of individual atomic steps
• Algebrarium evaluation: A controlled framework isolating single-step vs. multi-step reasoning
• **Trade-off discovmay sacrifice performance on specific skills to optimize global reward

The Central Question

When a model trained with RLVR solves a complex multi-step reasoning problem it could not solve before, what actually changed?

Two competing hypotheses:

1. New skills: RLVR teaches the model fundamentally new reasoning capabilities
2. Sharper primitives: RLVR increases the reliability of existing atomic reasoning steps, making their composition more likely to succeed

This paper provides strong evidence for hypothesis 2.

Probabilistic Framework

Atomic Steps and Composite Tasks

Define a composite reasoning task $T$ as aquence of $n$ atomic steps:

$T = s_1 \circ s_2 \circ \cdots \circ s_n$

Each atomic step $s_i$ has a probability $p_i$ of being executed correctly by the model. The probability of successfully completing the composite task is approximately:

$P(T) \approx \prod_{i=1}^{n} p_i$

This multiplicative relationship means that even small improvements in atomic step probabilities can yield large improvements in composite task success:

If each $p_i$ increases from 0.7 to 0.9, a 5-step task goes from $0.7^5 = 0.168$ to $0.9^5 = 0.590$ — a 3.5x improvement.

The Sharpening Effect

RLVR “sharpens” the model’s policy by increasing atomic step probabilities:

$p_i^{\text{after RLVR}} > p_i^{\text{before RLVR}}$

This sharpening is sufficient to explain the emergence of complex reasoning without invoking new capabilities. The model always “knew” how to perform each step; RLVR simply made each step more reliable.

The Algebrarium Framework

Design

Algebrarium provides a controlled environment for testing the sharpening hypothesis:

• Atomic operations: Single-step algebraic operations (addition, multiplication, modular arithmetic, etc.)
• Composite tasks: Multi-step problems requiring chains of atomic operations
• Training: Models are trained with RLVR on single-step operations only
• Testing: Models are evaluated on unseen multi-step compositions

This design ensures that any improvement on composite tasks must come from improved atomic step execution, not from learning new compositional strategies.

Training Protocol

1. Pre-train a language model on standard data
2. Apply RLVR training using only single-step algebraic problems
3. Evaluate on multi-step problems never seen during RLVR training

Key Findings

Finding 1: RLVR Encourages Exploration

After RLVR training, models explore reasoning paths that were previously inaccessible:

• The model’s policy becomes less peaked on common but suboptimal paths
• Previously low-probability correct paths become accessible
• This exploration is not random but guided by the reward signal

Finding 2: Strong Correlation with Joint Probability

The composite task performance correlates strongly with the joint probability of atomic steps:

$\text{Pearson}(P(T), \prod_i p_i) \in [0.69, 0.96]$

This strong correlation across different task types and model sizes provides compelling evidence for the multiplicative probability model. The composite performance is well-predicted by simply multiplying the individual step probabilities.

Finding 3: Skill Trade-offs

RLVR can sacrifice performance on specific atomic skills to maximize overall reward:

• If improving skill $A$ by $\delta$ costs skill $B$ by $\epsilon$, RLVR will make this trade if the net effect on expected reward is positive
• This explains why RLVR-trained models sometimes perform worse on specific subtasks despite overall improvement
• The optimization is global, not local to individual skills

Finding 4: No Evidence of New Capabilities

Across all experiments, the authors find no evidence that RLVR creates genuinely new reasoning capabilities:

• All atomic operations the model performs after RLVR were already present (at lower probability) before
• The improvement is quantitative (higher probability) not qualitative (new operations)
• Multi-step success is fully explained by atomic step probability improvements

Analysis

Why Sharpening Is Sufficient

Consider a 5-step reasoning chain where each step has probability 0.8 before RLVR:

$P(\text{success}) = 0.8^5 = 0.328$

After RLVR sharpens each step to 0.95:

$P(\text{success}) = 0.95^5 = 0.774$

The model goes from failing most of the time to succeeding most of the time, yet no new capability was added — each step was already possible.

Implications for Scaling

The multiplicative model suggests that:

• Longer reasoning chains are exponentially harder (probability drops exponentially with chain length)
• Diminishing returns: Once atomic steps are near-perfect, further RLVR training yields little improvement
• Bottleneck identification: The weakest atomic step limits composite performance most

Connection to Exploration in RL

The exploration finding connects to classical RL theory:

• RLVR’s reward signal guides the model toward correct reasoning paths
• Paths that were theoretically possible but practically never sampled become accessible
• This is exploration in reasoning space, not just action space

Significance

This paper provides a principled answer to a fundamental question about RLVR and reasoning emergence. The probabilistic framework elegantly explains why RLVR appears to create new reasoning abilities when it is actually sharpening existing ones. The strong empirical correlation between atomic step probabilities and composite performance (Pearson 0.69-0.96) provides compelling quantitative support. This understanding has practical implications: to improve complex reasoning, focus on identifying and strengthening the weakest atomic steps rather than training on complex tasks directly.