Improving Parametric Knowledge Access in Reasoning Language Models

Paper: 2602.22193 Authors: Melody Ma, John Hewitt Categories: cs.CL

Abstract

This research investigates how reasoning language models access world knowledge stored in their parameters. While these models excel at mathematical reasoning through reinforcement learning, they underperform when retrieving factual knowledge. The authors discover that a simple “think step-by-step” prompt significantly improves knowledge recall but not mathematical performance, revealing a gap in how models apply reasoning to different domains. They propose training models via reinforcement learning on world-knowledge question answering tasks, using TriviaQA as the primary training dataset. This approach yields a 9.9% improvement on TriviaQA and transfers well to other benchmarks: Natural Questions (+4.2%), HotpotQA (+2.1%), SimpleQA (+0.6%), and StrategyQA (+3.0%).

Key Contributions

• Empirical demonstration that reasoning models generate suboptimal reasoning traces for parametric knowledge access by default
• Evidence that simple prompting interventions (“think step-by-step”) improve knowledge recall but not mathematical reasoning, suggesting domain-specific optimization gaps
• A reinforcement learning framework for training models to reason over their parametric knowledge using verifiable world-knowledge QA rewards
• Significant performance improvements across multiple knowledge-intensive benchmarks with strong transfer learning effects
• Analysis showing that reasoning models are systematically under-optimized for accessing their own stored knowledge

Methodology and Training Approach

The researchers employ a targeted reinforcement learning strategy to enhance parametric knowledge access. Unlike traditional reasoning model training that focuses on mathematical tasks, this work uses world-knowledge question answering as both the training objective and the reward signal. The training process leverages TriviaQA, a dataset containing factual questions with verifiable answers, allowing the model to receive clear feedback on whether its reasoning process successfully retrieves correct information.

The key insight is that reasoning traces beneficial for knowledge retrieval differ from those optimized for mathematical problem-solving. For instance, recalling that Canberra is Australia’s capital might benefit from reasoning through concepts like “purpose-built capitals” or “major Australian cities,” creating a structured path to the correct answer. The reinforcement learning objective encourages the model to develop such knowledge-accessing reasoning patterns.

The training methodology maintains the model’s existing capabilities while specifically enhancing its ability to reason through factual recall. This targeted approach avoids the common pitfall of catastrophic forgetting, where training on new tasks degrades performance on previously learned ones.

Experimental Results and Analysis

The experimental results reveal substantial improvements across diverse knowledge-intensive benchmarks. On TriviaQA, the primary training dataset, the model achieves a 9.9% performance gain, demonstrating effective learning of knowledge-accessing reasoning patterns. More importantly, these improvements transfer to unseen datasets: Natural Questions shows a 4.2% increase, HotpotQA improves by 2.1%, and StrategyQA gains 3.0%.

The varying magnitudes of improvement across datasets provide insights into the nature of knowledge access reasoning. Larger gains on Natural Questions and StrategyQA suggest these tasks particularly benefit from structured reasoning over parametric knowledge. The more modest improvement on SimpleQA (0.6%) indicates that some knowledge access patterns may be dataset-specific or that SimpleQA requires different reasoning strategies.

Critically, the “think step-by-step” prompting experiment establishes a baseline showing that models possess latent capabilities for better knowledge reasoning but don’t employ them by default. This finding validates the core hypothesis: reasoning models need explicit optimization for parametric knowledge access, as their default training (focused on mathematical reasoning) doesn’t naturally generalize to this domain.

Implications for Language Model Development

This research has significant implications for how we train and deploy reasoning language models. First, it challenges the assumption that reasoning capabilities developed for one domain (mathematics) automatically transfer to others (knowledge retrieval). The results suggest that reasoning is not a monolithic skill but rather a collection of domain-specific strategies that require targeted optimization.

Second, the work demonstrates that parametric knowledge—information stored in model weights—can be more effectively accessed through learned reasoning patterns. This is particularly relevant as language models grow larger and store more world knowledge. Rather than relying solely on retrieval-augmented generation or external knowledge bases, models can be trained to better utilize their internal knowledge through structured reasoning.

Third, the strong transfer learning effects across multiple benchmarks suggest that knowledge-accessing reasoning patterns are somewhat generalizable. Training on one knowledge-intensive dataset (TriviaQA) improves performance on others, indicating that the model learns broadly applicable strategies for navigating its parametric knowledge rather than dataset-specific tricks.

Finally, this research opens questions about the optimal balance between different types of reasoning training. Future work might explore curriculum learning approaches that systematically develop both mathematical and knowledge-accessing reasoning, or investigate whether other domains (commonsense reasoning, causal inference) similarly require specialized reasoning optimization.

Takeaways

1. Reasoning language models are systematically under-optimized for accessing their parametric knowledge, despite strong performance on mathematical reasoning tasks
2. Simple prompting interventions reveal latent capabilities, but explicit reinforcement learning training is needed for consistent improvement
3. Training on world-knowledge QA tasks yields substantial gains (9.9% on TriviaQA) with strong transfer to other knowledge-intensive benchmarks
4. Reasoning strategies are domain-specific: what works for mathematics doesn’t automatically work for knowledge retrieval
5. The gap between prompted and default performance suggests significant room for improvement in how reasoning models are trained and deployed