Paper: 2608.28560 Authors: Emily Cheng, Ryan Cotterell Categories: cs.CL
The Gap
Can a listener recover what a speaker means from the form of an utterance alone? This is the question at the centre of a familiar debate — whether scaling text and supervision can yield something that understands meaning — and it has usually been argued from examples. Here it is answered information-theoretically.
The framing is what makes the result general. The listener is any featurizer of text, which explicitly includes the hidden states of contemporary large language models. So the bound is not a claim about a particular architecture or training regime; it is a ceiling that applies to any representation of an utterance, including the best one we currently have.
That matters because the usual argument in this area moves in the wrong direction. Evidence that a model handles some meaning-sensitive task is taken as evidence that the representation carries meaning. A bound says something different and stronger: how much meaning is recoverable from form at all, before any question of which model or how much data arises.
THE QUESTION, AND HOW IT IS USUALLY ARGUED
"can a listener recover what a speaker means
from the form of an utterance alone?"
|
+-- usual approach: EXAMPLES
| "the model gets this meaning-sensitive
| task right, therefore the representation
| carries meaning"
|
+-- this paper: INFORMATION-THEORETIC BOUND
the listener is ANY featurizer of text,
INCLUDING LLM HIDDEN STATES
|
v
[WHY THE FRAMING MATTERS]
the bound is a property of the TASK, not of a
model or a training regime
-> it applies to any representation of an
utterance, including the best available
The Increment
One sentence: Before this paper, the limits of learning meaning from text were argued from examples; after it, an information-theoretic upper bound applies to any decoder of any text representation, and decomposes the unavoidable uncertainty into an irreducible part and a part only context can resolve.
Core Mechanism
The construction is a joint distribution over three things — meanings, contexts, and utterances — which is what allows the question to be posed precisely. Given that distribution, the authors derive upper bounds on the probability that a decoder recovers a speaker’s intended meaning from a representation of the utterance. Upper bounds are the right instrument here: a lower bound would say some model can do at least this well; the ceiling is what constrains the debate.
The bound’s structure is the substance, and it is a decomposition. The uncertainty that form leaves about meaning splits into two parts:
- An irreducible component — uncertainty that no amount of information could remove, because the mapping from meaning to form is not injective in those cases.
- A component that only extralinguistic context can resolve — never the utterance alone. Pointing at something is not in the sentence. Who is being addressed is not in the sentence. The shared situation that made the utterance appropriate is not in the sentence.
This second part is where the negative claim gets its teeth. It is not that current models lack context, or that more text would help; it is that a class of the relevant information is, by construction, absent from the utterance. Reading the utterance more carefully — or training on more utterances — cannot recover information the utterance does not carry.
The generality conditions are stated carefully. Because these quantities are intrinsic to language, no representation, however much text or supervision produced it, can surpass them. That phrase does the load-bearing work: it forecloses the natural response that a sufficiently large model would learn a better featurizer and beat the bound. And the bounds hold whether the space of meanings is discrete or continuous — so the result is not an artefact of a convenient finite label set.
Empirical support comes from three settings chosen to exercise different parts of the theory: artificial languages (where the joint distribution is fully controlled), Mandarin zero-pronoun resolution (a case where the missing information is genuinely contextual — Chinese frequently omits subjects and objects that the discourse, not the sentence, determines), and color reference (a continuous meaning space, where the boundary between what the words fix and what the situation fixes is unusually vivid).
THE CONSTRUCTION
joint distribution over
MEANINGS x CONTEXTS x UTTERANCES
|
v
upper bound on P(decoder recovers intended meaning
from a REPRESENTATION OF THE UTTERANCE)
|
v
THE UNCERTAINTY THAT FORM LEAVES SPLITS INTO:
[ IRREDUCIBLE ]
meaning-to-form is not injective in these cases
-> no information could remove this
(an upper bound is the right instrument: a lower
bound would only say some model does at least so well)
[ EXTRALINGUISTIC-CONTEXT-ONLY ]
what is being pointed at -- not in the sentence
who is being addressed -- not in the sentence
the shared situation -- not in the sentence
-> reading the utterance MORE CAREFULLY cannot help,
nor can training on more utterances
|
v
GENERALITY: intrinsic to language, so NO representation,
however much text or supervision produced it, can surpass
these bounds -- discrete or continuous meaning space
EVIDENCE: artificial languages | Mandarin zero-pronoun
resolution | color reference
Think of it as the difference between a detailed receipt and knowing what the customer wanted. A receipt is a complete, faithful record of a transaction’s form — every item, every price. Studying it harder tells you exactly what was bought and no more. It cannot tell you why, whether it was a gift, or whether the buyer would have preferred something else, because none of that was ever on the paper. Training a better receipt-reader, or collecting a billion more receipts, improves your reading of receipts; it does not add the intention. The Mandarin zero-pronoun case is the receipt with the item name omitted: the omission is grammatical and normal, and filling it in requires knowing the conversation, which is exactly what the sentence does not contain.
Key Concepts
- Bounding any featurizer, including LLM hidden states: the result applies to any representation of an utterance. It is what makes this a statement about the task rather than about a model, and it forecloses “a bigger model would do better”.
- The two-part decomposition of form-uncertainty: irreducible uncertainty plus uncertainty only extralinguistic context can resolve. The second part is the substantive claim — a class of the relevant information is absent from the utterance by construction, not by data scarcity.
- Intrinsic to language: the phrase that closes the escape route. If the bound derives from the joint distribution of language use rather than from a model class, then no amount of text, parameters or supervision changes it.
- An upper bound as the right instrument: a lower bound would demonstrate capability; a ceiling constrains what can be claimed. For a debate conducted mostly through examples, the ceiling is what is missing.
Framework Shift
Before (argued from examples):
"the model handles this meaning-sensitive task"
-> taken as evidence the representation carries meaning
-> the debate is about which model, how much data
-> "a bigger model would do better" stays available
After (bounded for any representation):
joint distribution over meanings, contexts, utterances
-> upper bound on recovering intent from form
-> splits into irreducible + context-only parts
-> intrinsic to language: no representation surpasses it
-> holds for discrete AND continuous meaning spaces
From asking which model recovers meaning best, to asking how much meaning form carries at all, the core shift is that part of what a speaker means is not in the utterance, and no amount of reading the utterance will change that.
Expert Assessment
Problem choice: Excellent, and unusually well judged. The question of whether text alone suffices for meaning is one of the field’s standing arguments, and it is normally conducted by trading examples. Reframing it as a property of the meaning-form joint distribution, and bounding any representation, is the version of the argument that can actually settle something.
Method maturity: The construction is the contribution and it is disciplined. Modelling meanings, contexts and utterances jointly is what makes the decomposition possible, and the two-part split is not a technical convenience — it separates what is impossible in principle from what is merely unavailable in practice, which is a distinction most discussions of this question blur. Stating generality for any featurizer, and for discrete and continuous meaning spaces, closes the two most obvious objections in advance.
Experimental integrity: The three empirical settings are well chosen rather than convenient: artificial languages give full control over the joint distribution, Mandarin zero-pronoun resolution is a case where the missing information is genuinely contextual, and colour reference exercises a continuous meaning space. These are illustrations of the theory rather than tests that could refute it, and the paper presents them as such. The honest limitation is that the bound is only as good as the model of language use it assumes — the joint distribution is an idealisation, and how tight the bound is for real language is a separate question from its existence.
Writing quality: The framing is stated in the first sentence and the decomposition is explained in terms of what each part means, which is what makes an abstract theory readable. Because the practical upshot is a limitation on what models can be built to do, a short passage drawing out the consequences — which tasks should be expected to have a context-free ceiling, and which should not — would make the result considerably more actionable for readers deciding what to attempt.
Verdict: strong accept — it converts a debate conducted by anecdote into a bound that applies to any text representation, and the decomposition identifies exactly which part of the difficulty is not addressable by more data.
Takeaways
- Distinguish information absent from the input from information a model failed to extract. A bound over any featurizer is the way to tell which one you are dealing with.
- Expect a context-free ceiling on tasks where the utterance underdetermines the meaning. Mandarin-style argument dropping and deixis are cases where more text cannot substitute for the situation.
- Read an upper bound as a constraint on claims, and a lower bound as a demonstration of capability. The former is what settles disagreements conducted through examples.
- Check whether a stated limitation is about a model class or about the problem. If it derives from the joint distribution of the task, no amount of scale or supervision removes it.
论文: 2608.28560 作者: Emily Cheng, Ryan Cotterell 分类: cs.CL
缺口
一个听者能否仅凭话语的形式，恢复出说话人想表达的意义？ 这是一个长期争论的核心问题——“只靠扩大文本与监督规模，能否得到某种真正理解意义的东西”——而它此前通常靠举例来论证。而本文用信息论给出了答案。
正是这个设定让结论具有普遍性。 这里的听者是任何文本特征提取器（featurizer），其中明确包含当代大语言模型的隐状态。因此这个界不是关于某个特定架构或训练方式的断言；它是一条天花板，适用于对一段话语的任何表示，包括我们目前拥有的最好的那种。
这一点之所以重要，是因为这一领域通常的论证方向是错的。 “模型在某个对意义敏感的任务上表现良好”，被当作”该表示携带了意义”的证据。而一个上界说的是另一件更强的事：从形式中究竟能恢复出多少意义——这个问题先于”哪个模型”和”多少数据”而成立。
问题，以及通常如何论证
"一个听者能否仅凭话语的形式，
恢复出说话人想表达的意义？"
|
+-- 通常的做法：举例
| "模型在这个对意义敏感的任务上做对了，
| 所以这个表示携带了意义"
|
+-- 本文：信息论意义上的「界」
听者是「任何」文本特征提取器，
「包括 LLM 的隐状态」
|
v
[为什么这个设定重要]
这个界是「任务」的属性，而不是某个模型
或某种训练方式的属性
-> 它适用于对一段话语的任何表示，
包括目前最好的那种
增量
一句话: 在这篇论文之前，“从文本中学习意义的极限”是靠举例来论证的；在这篇论文之后，一个信息论上界适用于对任何文本表示的任何解码器，并把不可避免的不确定性分解为”不可约”与”只能由语境解决”两部分。
核心机制
构造是建立在三样东西上的联合分布——意义、语境、话语——正是它让这个问题得以被精确地提出。在这个分布之下，作者推导出了**“解码器从话语的某个表示中恢复说话人意图”的概率上界**。上界在这里是正确的工具：下界只能说”某个模型至少能做到这么好”；而天花板才是约束这场争论的东西。
界的结构才是实质所在，而它是一个分解。形式对意义所留下的不确定性被拆成两部分：
- 不可约的部分——在这些情形下，不存在任何数量的信息能消除它，因为从意义到形式的映射并非单射。
- 只能由语言外语境解决的部分——永远不能仅靠话语。指着什么，不在句子里；在跟谁说话，不在句子里；让这句话变得恰当的那个共享情境，不在句子里。
第二部分才是这个否定性结论真正有力度的地方。 它不是说”当前模型缺乏语境”，也不是说”更多文本会有帮助”；而是说：有一类相关信息，在构造上就不在话语之中。更仔细地读这句话——或者用更多话语去训练——都无法恢复话语本身并不携带的信息。
普遍性条件被谨慎地陈述。 由于这些量是语言内在的，任何表示——无论它由多少文本或监督产生——都无法越过它们。这句话承担了承重的工作：它预先堵住了那个很自然的反驳——“一个足够大的模型会学到更好的特征提取器，从而击败这个界”。而这些界在意义空间是离散或连续时都成立——因此这个结果不是某个方便的有限标签集的产物。
实证支持来自三个被挑选来检验理论不同部分的设定：人工语言（联合分布完全可控）、汉语零代词消解（缺失的信息确实是语境性的——中文经常省略由篇章而非句子决定的主语和宾语），以及颜色指称（一个连续的意义空间，词语固定了什么、情境固定了什么的边界在这里格外清晰）。
构造
在「意义 × 语境 × 话语」上的联合分布
|
v
上界：P(解码器从「话语的表示」中恢复出意图)
|
v
「形式对意义留下的不确定性」拆成：
[ 不可约 ]
在这些情形下，意义到形式的映射不是单射
-> 没有任何信息能消除它
（上界才是正确的工具：下界只能说
"某个模型至少能做到这么好"）
[ 只能由语言外语境解决 ]
指着什么 —— 不在句子里
在跟谁说话 —— 不在句子里
共享情境 —— 不在句子里
-> 把这句话读得「更仔细」没有用，
用更多话语训练也没有用
|
v
普遍性：这些量是语言内在的，因此「任何表示」——
无论由多少文本或监督产生——都无法越过；
意义空间无论离散还是连续都成立
证据：人工语言 | 汉语零代词消解 | 颜色指称
可以用**“一张详尽的购物小票，与「顾客当时想要什么」之间的差别”来理解这件事： 小票是一次交易在形式上完整而忠实的记录——每一件商品、每一个价格。再仔细地研究它，你得到的也只是”买了什么”，不会更多。它无法告诉你为什么买、是否是要送人、或者买家本来是否更想要别的东西——因为这些东西从来没有被写在纸上**。训练一个更好的读票器、或者收集十亿张新小票，改善的是你读小票的能力，而不是多出那份意图。 汉语零代词消解的案例，就是一张省略了商品名的小票：这种省略在语法上是正常且普遍的，而要把它补上，需要知道当时的对话——而那恰恰是句子所不包含的。
关键概念
- 对任何特征提取器给出界，包括 LLM 隐状态: 结果适用于对一段话语的任何表示。正是它让这条结论成为关于任务、而不是关于某个模型的陈述，并预先堵住了”更大的模型会做得更好”。
- 把”形式的不确定性”拆成两部分: 不可约的不确定性，加上只能由语言外语境解决的不确定性。第二部分才是实质主张——有一类相关信息在构造上就不在话语里，而不是”数据不够”。
- 语言内在（intrinsic to language）: 那个封死退路的关键限定。如果这个界源自语言使用的联合分布、而不是源自某个模型类，那么再多的文本、参数或监督都无法改变它。
- 以上界作为正确的工具: 下界展示能力，天花板约束你能声称什么。对一场主要靠举例进行的争论而言，天花板正是缺失的那一半。
框架转变
之前（靠举例论证）：
"模型在这个对意义敏感的任务上做对了"
-> 被当作「该表示携带了意义」的证据
-> 争论围绕"哪个模型、多少数据"
-> "更大的模型会更好"始终是一个可用选项
之后（对任何表示都给出界）：
在意义、语境、话语上的联合分布
-> 从形式中恢复意图的概率上界
-> 拆为「不可约」+「只能靠语境」两部分
-> 语言内在：任何表示都无法越过
-> 对离散与连续的意义空间都成立
从”问哪个模型最能恢复意义”，转变为”问形式究竟携带了多少意义”，核心转变在于：说话人想表达的东西，有一部分并不在话语里——而再怎样读这句话，也不会改变这一点。
专家评审
选题眼光: 极好，而且判断得格外到位。 “仅靠文本是否足以承载意义”是这个领域长期存在的争论之一，而它通常以交换举例的方式进行。把它重新表述为”意义—形式联合分布”的一个性质，并对任何表示给出界，才是这场争论中能够真正定论的那个版本。
方法成熟度: 构造就是贡献，而且它是有纪律的。 把意义、语境、话语放在一起建模，才使那个分解成为可能；而两部分拆分并非技术上的便利——它把”原则上不可能”与”实践上暂时拿不到”区分开来，而这个区分正是大多数相关讨论所模糊掉的。把普遍性陈述为”对任何特征提取器成立”、并覆盖离散与连续意义空间，等于提前封住了两个最显然的反驳。
实验诚意: 三个实证设定是被挑选的，而不是顺手拿来的：人工语言提供对联合分布的完全控制；汉语零代词消解是”缺失信息确实是语境性”的案例；颜色指称则检验连续的意义空间。它们是理论的例证，而不是能证伪它的检验，论文也这样呈现。 诚实的局限在于：这个界的强度取决于它对语言使用的建模——那个联合分布是一种理想化，而”对真实语言而言这个界有多紧”，是一个与该界是否存在相互独立的问题。
写作功力: 设定在第一句话里给出，分解则按每一部分”意味着什么”来解释——这正是让一套抽象理论可读的原因。 由于实际的落点是”模型能被造成做什么”的限制，若能补一小段把后果拉出来——哪些任务理应存在一个”无上下文的天花板”、哪些不该——会让结果对”正在决定该尝试什么”的读者有用得多。
判决: 强接收（Strong Accept） — 它把一场靠轶事进行的争论，转化为一个适用于任何文本表示的界；而那个分解准确地指出了”难处的哪一部分”不是靠更多数据能解决的。
要点总结
- 区分”输入中本就不存在的信息”与”模型没能提取出来的信息”。一个覆盖任何特征提取器的界，正是判断你面对的是哪一种的方法。
- 对那些”话语不足以确定意义”的任务，预期存在一个无上下文的天花板。汉语式的论元省略与指示词，就是”更多文本无法替代情境”的例子。
- 把上界读作对主张的约束，把下界读作对能力的展示。前者才是终结”靠举例进行的争论”的东西。
- 检查一条被陈述的限制，是关于模型类的，还是关于问题本身的。如果它源自任务的联合分布，那么再多的规模或监督都无法移除它。