AdaSR: Streaming Reasoning That Learns When to Think — A Hierarchical RL Approach

Paper: 2606.14694 Authors: Junlong Tong, Wenqi Xu, Yingqi Fan, Anhao Zhao, Xuan Lu, Yang Tan, Xiaoyu Shen Categories: cs.CL

The Gap

Existing large reasoning models operate on a “read-then-think” paradigm: they ingest the complete input, reason over a static context, and only then produce an answer. This works for text, but fails for dynamic streams like audio or video where information trickles in over time. Recent streaming reasoning methods (e.g., StreamLLM, LLM in a Flash) try to think while reading, but they rely on supervised imitation learning from pre-constructed trajectories — essentially memorizing when to pause and reason. That’s brittle: it cannot adapt to varying input rates, task difficulty, or latency budgets.

AdaSR fills this gap by framing streaming reasoning as a sequential decision problem and optimizing it with reinforcement learning. The core insight: the model should learn when to think (streaming reasoning phase) and how much to compute (deep reasoning phase), guided by a reward that penalizes latency and rewards accuracy.

[Logic Topology: Gap to Conclusion]

Problem: Static reasoning fails on dynamic streams
    |
    v
Assumption: RL can teach a model to allocate computation adaptively during a stream
    |
    v
Method: AdaSR + HRPO (Hierarchical Relative Policy Optimization)
    |   - Two-phase reasoning (streaming + deep)
    |   - Fine-grained advantage assignment per phase
    |   - Reward: format + accuracy + adaptive thinking
    |
    v
Evidence: Experiments show better accuracy / efficiency / latency trade-off vs SFT baselines
    |
    v
Conclusion: Adaptive streaming reasoning is achievable, and RL is the right tool

The Increment

One sentence: Before this paper, streaming reasoning models imitated fixed think-when-to-think behaviors; after this paper, models can learn adaptive compute allocation via hierarchical RL rewards.

Core Mechanism

AdaSR operates in two phases:

1. Streaming Reasoning Phase: As tokens of the input stream arrive one by one, the model produces partial answers (or continues its internal chain-of-thought) on the fly. At each step, it decides whether to output a reasoning token or wait for more stream data. The policy is optimized to minimize latency while maintaining correctness.

2. Deep Reasoning Phase: Once the stream ends, the model enters a final deliberation stage where it can revisit previous partial reasoning, integrate all information, and produce a final answer. This phase has no latency pressure; the model can allocate more computation as needed.

The training uses Hierarchical Relative Policy Optimization (HRPO). Standard PPO assigns a single sequence-level advantage uniformly to every token. HRPO splits the advantage into two parts: one for the streaming phase and one for the deep phase. This allows the reward signal to distinguish between “good streaming decisions” (e.g., outputting a correct partial answer early) and “good final deliberation decisions” (e.g., correcting an earlier mistake). On top of that, three reward terms are combined:

• Format reward: enforce valid reasoning protocol (e.g., output tags like …)
• Accuracy reward: final answer correctness
• Adaptive thinking reward: penalize excessive streaming-phase computation and reward low latency

[Internal Data Flow Diagram]

Input Stream (token by token):
  [token_1] -> [token_2] -> ... -> [token_n]

        |--- Streaming Reasoning Phase ---|
        v                                  v
  +------------+   +------------+   +------------+
  | Partial    |   | Partial    |   | Partial    |
  | Reasoning  |   | Reasoning  |   | Reasoning  |
  | Step 1     |   | Step 2     |   | Step N     |
  +------------+   +------------+   +------------+
                                          |
                                          | stream ends
                                          v
                                +--------------------+
                                | Deep Reasoning     |
                                | (Final Deliberation)|
                                +--------------------+
                                          |
                                          v
                                   Final Answer
                                          |
    Training: HRPO applies advantage per phase, not per token.
    Advantage = baseline(token) - reward(token) but grouped by phase.

Structural metaphor: The Improvising Chef

Think of a live cooking show where ingredients arrive one by one. The chef (model) must start cooking while waiting for the next ingredient. The chef has a sous chef (streaming reasoning) who makes preliminary decisions — “I’ll chop the tomatoes now because they’re here, even though the basil hasn’t arrived yet.” The sous chef can also send out partial dishes (partial answers) if they’re good enough.

But as soon as the last ingredient arrives, the head chef (deep reasoning) takes over. She reviews the sous chef’s work, adjusts seasoning, and finalizes the dish. The restaurant has a critic (reward function) who judges:

• “You didn’t follow the recipe format” → format penalty
• “The final dish tastes great” → accuracy reward
• “You took too long to serve” → latency penalty

The kitchen manager (HRPO) doesn’t give the same bonus to the sous chef and head chef. If the sous chef makes a clever anticipation that saves time, she gets a special streaming-phase advantage. If the head chef fixes a mistake the sous chef made, the advantage goes to deep reasoning. This targeted feedback lets each phase learn its own optimal behavior.

Key Concepts

• Hierarchical Advantage Assignment: In RL, an “advantage” measures how much better an action was compared to the average. Standard PPO computes one advantage for the whole sequence. HRPO splits it per phase. Example: a streaming-action that outputs a correct partial answer early gets high streaming-phase advantage, even if the final answer is wrong due to later confusion. This prevents the streaming phase from being penalized for downstream errors it didn’t cause.

• Adaptive Thinking Reward: A reward term that encourages the model to minimize computation during the streaming phase without hurting accuracy. It’s computed as a function of latency (e.g., number of tokens output before stream ends). This is essential for real-time applications where early partial answers are valuable.

Framework Shift

Before (mainstream approach):        After (this paper):

Input stream                         Input stream
    |                                    |
    v                                    v
[Wait for complete input]          [Streaming Reasoning Phase]
    |                                    |  (think while reading)
    v                                    v
[Static Reasoning]                  [Partial decisions / outputs]
    |                                    |
    v                                    | stream ends
[Final Answer]                           v
                                   [Deep Reasoning Phase]
                                        |
                                        v
                                   [Final Answer]

Both phases use same model, but optimized separately via HRPO.
Key difference: reasoning is decomposed into two temporally distinct stages,
each with its own reward structure. No more one-size-fits-all advantage.

One sentence: From static read-then-think to adaptive two-phase reasoning with phase-specific RL rewards, the core shift is treating streaming decisions as learnable, not imitable.

Expert Assessment

Problem choice: Real gap. Streaming AI is a growing area (live captioning, real-time translation, robotics) and most reasoning models are built for offline text. The paper sits at a timely intersection of streaming processing and reinforcement learning.

Method maturity: Clever insight — the hierarchical advantage decomposition is elegant and likely to be extended. However, the approach is fairly complex: training with RL, three reward terms, two-phase architecture. It’s not brute force, but it’s not minimal either. A simpler approach could be to use a learned router for when to think, but that might not capture the full optimization. So the complexity is justified.

Experimental integrity: The baselines are SFT models fine-tuned on pre-collected trajectories. That’s fair — it shows RL beats imitation. But I’d want to see comparisons with other RL methods (e.g., standard PPO without hierarchy) to isolate the benefit of HRPO. Also, the paper doesn’t discuss training stability or reward shaping sensitivity. Numbers seem plausible but no dramatic SotA claims — okay.

Writing quality: The main paper is solid but the reward design section is thin. It says “format, accuracy, adaptive thinking rewards” but doesn’t give exact formulas or hyperparameter choices. A more detailed ablation would elevate it.

Verdict: weak accept — interesting direction with a clean RL formulation, but needs stronger baselines and more transparency in reward engineering.

Takeaways

• Hierarchical advantage assignment can be applied to any multi-stage decision process: code generation (plan then write), robot control (explore then exploit), or multi-turn dialogue (short-term vs long-term goals). The idea is to reward each stage for its own contributions, not homogenize credit.
• The adaptive thinking reward is a practical proxy for latency. Practitioners building real-time LLM applications can adopt this directly to optimize speed vs quality trade-offs.
• The two-phase reasoning pattern (streaming + deep) is a reusable architecture: you can plug any base model into this framework and train with HRPO.