SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions

Paper: 2604.08477 Authors: Ashima Suvarna, Kendrick Phan, Mehrab Beikzadeh, Hritik Bansal, Saadia Gabriel (UCLA) Categories: cs.AI

The RLVR Ceiling

Reinforcement Learning with Verifiable Rewards (RLVR) has transformed reasoning in formal domains—mathematics and code—where ground truth is automatically verifiable. Models trained on MATH competitions and CodeForces now achieve +50% gains on benchmarks like AIME24. But expose those same models to general reasoning tasks requiring causal inference, temporal understanding, or pragmatic reasoning, and performance drops by −8% on Big-Bench Extra Hard (BBEH).

The problem isn’t the algorithm. It’s the data. RLVR needs high-quality, verifiable training examples that span diverse reasoning skills. STEM domains have this in abundance: human-verified competition problems, executable code tests, symbolic proofs. General reasoning doesn’t. Scraping the web for “domain-specific” QA pairs (science, business, history) doesn’t help—models score well on MMLU but still fail BBEH. The skills don’t transfer.

SUPERNOVA makes a different observation: instruction-tuning datasets like SuperNI and FLAN already contain thousands of expert-annotated tasks encoding rich reasoning patterns—event understanding, question generation, object counting, temporal graphs. They just weren’t designed for RLVR. Many tasks are open-ended (hard to verify), not all tasks elicit strong reasoning, and the principles for curating RLVR data from instructions remain unexplored.

The Framework

SUPERNOVA is a multi-stage pipeline for curating RLVR data from instruction datasets. It explores three design decisions through 100+ compute-matched RL experiments:

1. Task Selection

Starting with 83 candidate tasks from SuperNI, SUPERNOVA reformats open-ended instructions into verifiable formats (e.g., multiple-choice questions). For each task, it samples 8 rollouts from the model, computes per-question win-rate, and removes questions that are trivial (win-rate=1) or impossible (win-rate=0).

Tasks are ranked by utility: train the model on task Dk, evaluate on BBEH’s 23 sub-tasks, measure performance. The ranking reveals that source task selection has dramatic impact on downstream reasoning. Not all instruction tasks are created equal.

2. Task Mixing

Two strategies for combining top-ranked tasks:

• Macro Mixing: Select N tasks with highest average performance across all BBEH sub-tasks
• Micro Mixing: For each BBEH sub-task, select the top N tasks that improve it specifically, then take the union

Micro mixing consistently outperforms macro. Different reasoning skills benefit from different source tasks. A single “best on average” mixture leaves performance on the table.

3. Data Interventions

Can synthetic augmentations improve data quality? SUPERNOVA tests interventions that increase question difficulty while preserving verifiable answers: long-context dependencies, anti-prior prompts, needle-in-haystack constraints, multi-hop reasoning chains.

Surprising result: under a fixed training budget, augmenting original data with these interventions does not improve performance. The base instruction data, when properly selected and mixed, is already sufficient.

Results

SUPERNOVA curates 25K RLVR samples and trains Qwen3 models (0.6B–4B) for 5000 RL steps using GRPO.

BBEH-test (general reasoning):

• Qwen3-4B baseline: pass@1 = 15.4%, pass@8 = 24.5%
• SUPERNOVA-4B: pass@1 = 19.9% (+29.4%), pass@8 = 35.0% (+42.9%)
• Relative improvement of up to 52.8% on BBEH across model sizes

Generalization to other benchmarks:

• BBH, MMLU-Pro, Zebralogic: +12.3% average improvement on pass@8
• SUPERNOVA-4B outperforms the larger Qwen3-8B by 8.2% on pass@8

Test-time compute scaling: Performance improves from pass@1 to pass@8, indicating the model explores more effectively during reasoning rather than memorizing answers.

Key Insights

1. Task selection is non-trivial: Training on the “wrong” instruction tasks can hurt general reasoning. Semantic similarity to the target benchmark is not predictive—empirical utility ranking via training is necessary.

2. Micro mixing beats macro mixing: Per-sub-task selection outperforms average-based selection. General reasoning is not monolithic; different skills require different training signals.

3. Quality over augmentation: Under fixed compute, carefully selected base data outperforms augmented data. The reasoning patterns in expert-annotated instructions are already rich—the bottleneck is selection, not synthesis.

4. RLVR extends beyond STEM: With principled data curation, RLVR can improve general reasoning capabilities. The key is repurposing existing human-annotated resources rather than generating new synthetic data or scraping the web.

Implications

SUPERNOVA demonstrates that the instruction-tuning datasets we already have—SuperNI, FLAN, and similar resources—contain latent reasoning structure that RLVR can exploit. The challenge is not data scarcity but data curation: identifying which tasks transfer, how to mix them, and when to stop adding complexity.

For practitioners: if you’re extending RLVR to a new domain, start with existing instruction datasets. Reformat for verifiability, rank by empirical utility, mix per-target-skill, and resist the urge to augment before you’ve exhausted selection.

For researchers: the 100+ ablation experiments in SUPERNOVA provide a template for principled data curation. The finding that micro mixing outperforms macro mixing suggests that reasoning is compositional at the skill level, not the task level—a hypothesis worth exploring in other domains.