Verifier-Backed Hard Problem Generation for Mathematical Reasoning

Paper: 2605.06660 Authors: Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao Categories: cs.LG, cs.CL

Abstract

Large Language Models (LLMs) demonstrate strong capabilities in solving scientific and mathematical problems, yet they struggle to produce valid, challenging, and novel problems—an essential component for advancing LLM training and enabling autonomous scientific research. This paper introduces VHG (Verifier-backed Hard problem Generation), a framework built upon three-party self-play that integrates an independent verifier into the conventional setter-solver duality. By constraining the setter’s reward to be jointly determined by problem validity (evaluated by the verifier) and difficulty (assessed by the solver), VHG substantially outperforms baseline methods across multiple mathematical reasoning benchmarks.

Key Contributions

• Three-Party Self-Play Framework: Introduces a novel architecture that adds a verifier to the traditional setter-solver paradigm, preventing the generation of invalid problems that would otherwise collapse the training pipeline.

• Dual Verifier Variants: Implements both Hard (symbolic) and Soft (LLM-based) verifiers, enabling the framework to work across different domains—from formal mathematical tasks to general reasoning problems.

• Significant Performance Gains: Achieves 16.9%, 16.6%, and 21.4% improvements on indefinite integral benchmarks, and raises overall pass@1 accuracy from 56.8% to 69.0% on general math benchmarks (MATH, AMC, Minerva, Olympiad, AIME24-26).

• Cross-Model Generalization: Demonstrates that problems generated by smaller models (Qwen3-4B) can effectively challenge and improve larger models (Qwen3-8B, 14B, 32B), suggesting efficient scaling strategies.

Methodology

The VHG Framework

VHG extends standard self-play by introducing three key components:

1. Setter (Q): Generates problem-reference pairs (x, y*) where x is the problem and y* is the reference solution. Unlike traditional approaches that only generate problems, VHG requires the setter to also provide solutions, enabling verifiable validity checks.

2. Verifier (V): Validates the correctness of generated problem-reference pairs. Two variants are explored:

   • Hard Verifier: Uses symbolic verification mechanisms (e.g., SymPy for indefinite integrals) providing nearly 100% reliable verification
   • Soft Verifier: Uses LLMs to check the correctness of step-by-step problem generation, enabling broader domain applicability

3. Solver (S): Attempts to solve the generated problems multiple times, providing difficulty feedback based on empirical success rate.

Training Process

The framework operates through iterative cycles:

• The setter proposes problem-reference pairs
• The verifier gates validity (pass/fail)
• Only accepted pairs are scored by the solver for difficulty
• The setter is trained to maximize difficulty while maintaining validity
• The solver is trained on valid, challenging problems

This design eliminates reward hacking—the setter cannot achieve high rewards by generating invalid problems since the verifier filters them out before difficulty assessment.

Results

Indefinite Integral Tasks

On specialized mathematical tasks, VHG with hard verifier showed remarkable improvements:

• AntiderivBench Qualifier: +16.9% pass@1 accuracy
• AntiderivBench Competition: +16.6% pass@1 accuracy
• Integration Stress Test: +21.4% pass@1 accuracy

These results significantly outperform R-Zero, the previous state-of-the-art method.

General Mathematical Reasoning

With soft verifier on broader mathematical domains:

• Overall pass@1 accuracy improved from 56.8% to 69.0%
• Consistent gains across MATH, AMC, Minerva, Olympiad, and AIME24-26 benchmarks
• Substantial margin over baseline methods including naive self-play approaches

Key Findings

• Verifier is Critical: Ablation studies show that removing the verifier leads to rapid collapse of problem quality due to reward hacking
• Validity-Difficulty Trade-off: The framework successfully balances generating challenging problems while maintaining validity
• Scalability: Problems generated by 4B parameter models effectively train and challenge models up to 32B parameters

Takeaways

• Autonomous Problem Generation: VHG represents a significant step toward autonomous scientific research by enabling LLMs to generate their own training data without human expert involvement.

• Preventing Reward Hacking: The three-party design elegantly solves the reward hacking problem that plagues naive self-play approaches, where setters learn to generate invalid problems to maximize difficulty scores.

• Practical Training Paradigm: The framework provides a practical path for continuously increasing training data difficulty without costly human expert involvement, potentially enabling LLMs to surpass human-level performance.

• Broad Applicability: The soft verifier variant demonstrates that the approach can generalize beyond formal mathematical domains to areas where exact verification is impractical.

• Efficient Scaling: The finding that smaller models can generate useful training data for larger models suggests cost-effective strategies for improving large-scale models.