InftyThink+: Infinite-Horizon Reasoning via Reinforcement Learning

Paper: 2602.06960 Authors: Yuchen Yan, Liang Jiang, Jin Jiang, Shuaicheng Li, Zujie Wen, Zhiqiang Zhang, Jun Zhou, Jian Shao, Yueting Zhuang, Yongliang Shen Categories: cs.CL, cs.AI

Abstract

Large reasoning models using chain-of-thought scaling face three fundamental limitations: quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects. InftyThink+ proposes iterative reasoning with periodic summarization, optimized through reinforcement learning, to enable effective infinite-horizon reasoning.

Key Contributions

• Iterative reasoning framework: Breaks long reasoning chains into manageable segments with periodic summarization
• Two-stage training: Combines supervised learning initialization with trajectory-level reinforcement learning
• Strategic summarization: The model learns when and how to summarize, enabling continuation beyond context limits

The Problem

Standard chain-of-thought reasoning in LLMs faces three compounding issues:

1. Quadratic cost: Attention computation grows quadratically with sequence length
2. Context limits: Models have fixed context windows that cap reasoning depth
3. Lost-in-the-middle: Performance degrades as reasoning chains grow longer, with the model losing track of earlier steps

These limitations mean that simply “thinking longer” hits diminishing returns and eventually negative returns.

The Solution: Iterative Reasoning with Summarization

InftyThink+ breaks the reasoning process into iterations:

1. Reason: Generate a segment of reasoning within a manageable context window
2. Summarize: Compress the key insights and intermediate results
3. Continue: Start a new reasoning segment from the summary

The model learns when to summarize and what to preserve through RL training, making these decisions strategic rather than mechanical.

Two-Stage Training

• Stage 1 (SFT initialization): Supervised fine-tuning teaches the basic pattern of iterative reasoning with summarization
• Stage 2 (Trajectory-level RL): Reinforcement learning optimizes the full trajectory, teaching the model to make better summarization and continuation decisions

Results

• 21% accuracy improvement on AIME24 benchmark
• Outperforms conventional long chain-of-thought RL approaches
• Better generalization to out-of-distribution benchmarks
• Reduced inference latency compared to single long-chain approaches

Why This Matters

1. Breaks the context ceiling: Reasoning is no longer bounded by context window size
2. Efficient compute usage: Avoids quadratic blowup by working in segments
3. Learned compression: The model decides what information to carry forward
4. Practical scaling: Enables test-time compute scaling without architectural changes