Do LLMs Benefit From Their Own Words?

Paper: 2602.24287 Authors: Jenny Y. Huang, Leshem Choshen, Ramon Astudillo, Tamara Broderick, Jacob Andreas Categories: cs.CL, cs.AI

The Gap

Every major LLM interface—ChatGPT, Claude, Gemini—follows the same pattern: they keep the entire conversation history, including both user messages and the assistant’s own responses. This design seems obvious: how else would a model maintain coherence across turns? But this assumption has never been rigorously tested. The current frontier treats full conversation history as gospel, with no systematic investigation of whether models actually benefit from conditioning on their own prior outputs. The cost is real: context windows fill up fast, inference slows down, and memory consumption scales linearly with conversation length. Prior work has explored context compression and retrieval, but none have questioned whether the assistant’s own words deserve to be there at all.

The Increment

Before: LLMs condition on full conversation history by default, assuming their own responses are essential context. After: Selectively omitting assistant history maintains or improves quality while cutting context length by up to 10x.

Think of a conversation like a relay race. In the standard approach, each runner (turn) carries a baton that accumulates weight—every previous runner’s notes, strategies, and commentary get added to it. By the final lap, you’re lugging around a heavy bundle. This paper asks: what if each runner only needs to know what the coach (user) said, not what previous runners muttered to themselves? The user turns are the race strategy; the assistant turns are just execution artifacts.

The core mechanism is embarrassingly simple: strip out all assistant responses from the conversation history, keeping only user turns. When the model generates turn $t$, it sees user messages from turns $1, 2, \ldots, t$ but none of its own responses from turns $1, 2, \ldots, t-1$. The authors test this across real multi-turn conversations and find that 36.4% of prompts are self-contained (answerable from the current turn alone), and many follow-ups provide enough instruction that prior user context suffices. When user-turn-only prompting fails, it’s often because the user references something specific from an assistant response. When it wins, it’s because the model was over-conditioning on its own mistakes—what the authors call “context pollution.”

Key Concepts

Self-Contained Prompts: Imagine you’re debugging with a colleague. Sometimes you ask, “How do I reverse a list in Python?”—that question stands alone. Other times you say, “What about the edge case you mentioned?”—that requires prior context. A self-contained prompt is the first kind: it carries all necessary information within itself. The paper finds that 36.4% of real user turns are self-contained, meaning the model could answer them cold, without any conversation history. This matters because it sets a lower bound on how much history you can safely discard.

Context Pollution: Picture a student who keeps re-reading their own rough draft while writing the next paragraph. If the draft contains an error—say, they misremembered a date—they’ll keep reinforcing that error in subsequent paragraphs. Context pollution is when an LLM does this: it over-conditions on its own prior responses, propagating hallucinations, stylistic quirks, or mistakes across turns. For example, if the model hallucinates a fact in turn 2, and the user asks a follow-up in turn 3, the model might double down on the hallucination because it’s now part of the “established context.” Removing assistant history breaks this feedback loop.

User-Turn-Only Prompting: This is the experimental condition where you surgically remove all assistant responses from the conversation history, keeping only what the user said. It’s not about summarizing or compressing—it’s about asking whether the assistant’s own words are signal or noise. The surprising result: on a large fraction of turns, removing assistant history doesn’t hurt quality. When it does hurt, it’s usually because the user explicitly references something the assistant said (“Can you expand on that second point?”). When it helps, it’s because you’ve eliminated context pollution.

Expert Assessment

Problem significance: This is a high-impact problem. Every production LLM system deals with context length limits and inference costs. If you can safely omit half the conversation history, you double your effective context window and halve your memory footprint. The affected community is massive: anyone deploying conversational AI at scale.

Method maturity: This is closer to deployment-ready than proof-of-concept, but with caveats. The user-turn-only approach is trivial to implement—it’s a filtering step, not a new model architecture. The context-filtering variant (selectively omitting assistant turns based on heuristics) is more sophisticated but still practical. However, the paper doesn’t provide a robust decision rule for when to omit vs. retain assistant context. The heuristics are reasonable (check if the user references prior assistant output) but not rigorously validated. A production system would need more careful tuning.

Experimental rigor: The baselines are fair—standard full-context prompting is the right comparison. The dataset (real multi-turn conversations from WildChat) is representative of actual usage, which is a strength. However, the evaluation relies heavily on GPT-4 as a judge, which introduces potential bias: GPT-4 might prefer responses that match its own style, and user-turn-only prompting might produce outputs that differ stylistically even if they’re factually equivalent. The paper acknowledges this but doesn’t fully address it. Human evaluation on a subset would strengthen the claims. Also, the analysis of context pollution is mostly qualitative—more systematic measurement of error propagation would help.

Verdict: weak accept — Solid empirical work with immediate practical value, but the evaluation could be more rigorous and the decision rules for selective omission need refinement.

Takeaways

Audit your defaults: Just because everyone does something doesn’t mean it’s optimal. This paper’s core move—questioning an unexamined assumption—is transferable to any system design. What are you including in your context, training data, or feature set simply because “that’s how it’s done”? Run the ablation.

Error propagation is real: In any iterative or sequential system, mistakes compound. If your system’s output at step $t$ becomes input at step $t+1$, you need a mechanism to break error feedback loops. This applies to RL (where policy updates can amplify biases), data pipelines (where preprocessing errors propagate), and even code review (where early misunderstandings color later feedback).

Context is not always signal: More information can hurt if it’s noisy or misleading. This insight applies beyond LLMs: in retrieval systems, adding more documents doesn’t always improve answers. In feature engineering, more features can degrade model performance. The lesson: be surgical about what you condition on, and measure whether each piece of context actually helps.