DelTA: Teaching Language Models Which Tokens Actually Matter for Reasoning

Paper: 2605.21467 Authors: Kaiyi Zhang, Wei Wu, Yankai Lin Categories: cs.LG, cs.CL

The Gap

Reinforcement learning from verifiable rewards (RLVR) has become the standard way to improve LLM reasoning: generate multiple responses, check which ones are correct, and use policy gradients to increase probability of good responses while decreasing bad ones.

The problem: RLVR operates at the sequence level but updates token probabilities. When you tell the model “this entire response is good,” which tokens should get credit? Standard RLVR implicitly treats all tokens equally within advantage-weighted averaging. This creates a mismatch: high-frequency formatting tokens (like newlines, brackets, “Step 1:”) appear in both correct and incorrect responses, so they dominate the gradient signal. Meanwhile, the sparse tokens that actually distinguish correct reasoning from incorrect reasoning get drowned out.

Prior work either accepts this limitation or tries heuristic token-level reward shaping, which requires domain knowledge and doesn’t generalize. The gap: we need a principled way to identify which tokens are discriminative for correctness and amplify their influence in the policy gradient update.

Problem: RLVR rewards sequences, but updates tokens
   |
   v
Observation: Standard update = advantage-weighted centroid averaging
   |
   v
Issue: High-freq shared tokens dominate, sparse discriminative tokens diluted
   |
   v
Method: Learn token coefficients that amplify discriminative directions
   |
   v
Evidence: +3.26 points on math benchmarks (Qwen3-8B)
   |
   v
Conclusion: Discriminative token credit assignment improves RLVR efficiency

The Increment

One sentence: Before DelTA, RLVR treated all tokens in a response equally when assigning credit; after DelTA, the method automatically identifies and amplifies the sparse tokens that actually distinguish correct from incorrect reasoning.

Core Mechanism

DelTA reframes the RLVR policy gradient update as a linear discriminator operating over token-gradient vectors. In standard RLVR, you compute gradients for each token in each response, then average them weighted by advantages (positive for correct responses, negative for incorrect). This creates two centroids: one from correct-response tokens, one from incorrect-response tokens. The policy update direction is the difference between these centroids.

The problem is how these centroids are constructed. Advantage-weighted averaging gives equal structural weight to all tokens within a response. If “Step 1:” appears in 90% of both correct and incorrect responses, it contributes heavily to both centroids, making them less separated. DelTA introduces token-level coefficients that reweight the contribution of each token to its centroid. Tokens that appear similarly in both correct and incorrect responses get downweighted; tokens that are side-specific (appear mostly in correct or mostly in incorrect) get amplified.

These coefficients are learned by treating the problem as discriminative: estimate which tokens best separate the two sides. DelTA uses a simple logistic regression over token-gradient vectors, where the label is whether the response was correct. The learned coefficients then reweight a self-normalized RLVR surrogate objective. This makes the effective centroids more contrastive—they’re now built from the tokens that actually matter for distinguishing correctness, not just the tokens that appear most frequently.

Standard RLVR:
  Correct responses  -->  [avg all token grads]  -->  Centroid+
  Incorrect responses --> [avg all token grads]  -->  Centroid-
  Update direction = Centroid+ - Centroid-
  
  Problem: "Step 1:" dominates both centroids
  
DelTA:
  Correct responses  -->  [weighted avg: amplify discriminative tokens]  -->  Centroid+*
  Incorrect responses --> [weighted avg: downweight shared tokens]      -->  Centroid-*
  Update direction = Centroid+* - Centroid-*
  
  Result: Centroids separated by reasoning tokens, not formatting

Think of it like this: You’re a teacher grading math homework. Standard RLVR is like saying “this entire solution is correct, so reinforce everything equally”—including the student’s habit of writing “Problem 1:” at the top, which appears on every paper regardless of correctness. DelTA is like a teacher who’s seen thousands of papers and knows which parts of the solution actually indicate understanding. When you see “therefore x = 5” in a correct solution, you know that specific reasoning step matters. When you see “Step 1:” you know it’s just formatting. DelTA learns this discrimination automatically by comparing what appears in correct vs incorrect solutions, then adjusts how much credit each part gets. The student still learns to write complete solutions, but now the learning signal is concentrated on the reasoning steps that actually distinguish correct from incorrect work, not the boilerplate.

Key Concepts

• Token-gradient vector: For each token in a response, the policy gradient computes how much to adjust that token’s probability. This adjustment is a vector in parameter space—it points in the direction that would increase that token’s likelihood. In RLVR, you compute these vectors for every token in every sampled response. The key insight: these vectors encode not just “increase this token” but “increase this token in this context with this surrounding structure.” When you average token-gradient vectors from correct responses, you’re implicitly asking “what’s the common direction across all tokens in good responses?” DelTA’s observation is that this average is dominated by high-frequency tokens, so it reweights the average to emphasize discriminative tokens.

• Discriminative vs shared tokens: A discriminative token is one that appears much more frequently in correct responses than incorrect ones (or vice versa). For example, in math problems, ”= 0” might appear often in correct solutions but rarely in incorrect ones, making it discriminative. A shared token appears with similar frequency in both correct and incorrect responses—like “Step 1:” or newlines. Standard RLVR can’t distinguish these: both contribute to the gradient based on their frequency and advantage weight. DelTA explicitly estimates which tokens are discriminative by training a logistic classifier on token-gradient vectors, then uses the classifier’s coefficients to reweight the RLVR objective. This amplifies discriminative tokens and suppresses shared ones.

• Self-normalized surrogate objective: RLVR typically uses a clipped surrogate objective from PPO to prevent destructively large updates. DelTA modifies this by multiplying each token’s contribution by its learned coefficient, then renormalizing so the total “mass” of the objective stays constant. This is crucial: without renormalization, you’d just be scaling the entire gradient up or down. With renormalization, you’re redistributing credit—taking it away from shared tokens and giving it to discriminative ones. The self-normalization ensures that the overall update magnitude stays controlled (important for training stability) while the update direction becomes more contrastive.

Framework Shift

Before (standard RLVR):              After (DelTA):

Response-level rewards               Response-level rewards
        |                                    |
        v                                    v
  All tokens in response             Discriminative token
  weighted by advantage              coefficient estimation
        |                                    |
        v                                    v
  Centroid+ and Centroid-            Reweighted centroids
  (dominated by high-freq)           (amplify discriminative)
        |                                    |
        v                                    v
  Policy gradient update             Policy gradient update
  (diluted signal)                   (concentrated signal)

One sentence: From treating all tokens in a response as equally responsible for its correctness, to automatically identifying and amplifying the sparse tokens that actually distinguish correct from incorrect reasoning.

Expert Assessment

Problem choice: This is a real gap. The mismatch between sequence-level rewards and token-level updates has been known since the early days of RLHF, but most work either ignores it or applies domain-specific heuristics. The observation that high-frequency tokens dominate gradient centroids is not obvious from the math but becomes clear when you visualize token contributions. The problem sits at a productive intersection: it’s fundamental enough to matter across domains but specific enough to admit a clean solution.

Method maturity: The core idea—reweight token contributions to make centroids more discriminative—is elegant. The execution via logistic regression is almost too simple, which is actually a strength: it’s interpretable and doesn’t add much computational overhead. However, the paper could explore whether more sophisticated discriminators (e.g., attention-based) would help, or whether the linear assumption is a feature (prevents overfitting to spurious correlations). The self-normalization trick is clever and necessary, but the paper doesn’t deeply analyze what happens when discriminative tokens are very sparse—does the method degrade gracefully?

Experimental integrity: Baselines are fair and include recent strong methods (GRPO, ReST-EM). The gains are consistent across model scales and multiple math benchmarks, which is reassuring. The ablation studies are thorough—showing that both the discriminative coefficients and the self-normalization matter. One concern: the out-of-domain evaluation (GSM8K → MATH) shows smaller gains, suggesting the method might be somewhat task-specific despite claims of generality. The code generation results are weaker, which the authors acknowledge but don’t fully explain. More analysis of when and why DelTA helps would strengthen the claims.

Writing quality: The discriminator framing in Section 3 is the paper’s strongest contribution conceptually, but it’s buried in math. A clearer upfront explanation of “why do centroids get dominated by shared tokens” with a toy example would help. The related work section is thorough but could better position DelTA relative to token-level reward shaping methods. The experimental section is dense—some results could move to appendix to make room for deeper analysis of failure cases or limitations.

Verdict: weak accept — Solid idea with consistent empirical gains, but the method’s scope and failure modes need more exploration. The simplicity is a feature, not a bug, but the paper oversells generality given the mixed results on code generation.

Takeaways

Reweight, don’t rescale: When you have a noisy signal dominated by high-frequency components, don’t just filter or threshold—learn coefficients that redistribute mass from noisy to informative components while preserving total magnitude. This pattern transfers beyond RLVR: any time you’re averaging over heterogeneous contributions (e.g., attention weights, ensemble predictions, gradient aggregation), consider whether some components are “shared noise” that should be downweighted.

Discriminative framing for credit assignment: The insight that policy gradient updates implicitly construct a linear discriminator is useful beyond this paper. If you’re doing any form of contrastive learning or preference optimization, explicitly modeling what makes positive examples different from negative examples (rather than just “positive is good, negative is bad”) can sharpen the learning signal. The logistic regression approach is a minimal viable implementation—you could swap in other discriminators depending on your domain.

Self-normalization for stable reweighting: When you reweight components of a loss or gradient, renormalize to keep the total magnitude constant. This prevents the reweighting from accidentally amplifying or dampening the entire update, which would confound the effect you’re trying to measure. It’s a small trick but critical for making reweighting schemes work in practice without extensive hyperparameter tuning.