Paper: 2608.26036 Authors: Srimonti Dutta, Akshata Kishore Moharir Categories: cs.AI, cs.CL
The Gap
Answer accuracy is treated as the reliability signal for LLM data agents, and for structured-data tasks it is the wrong one. A benchmark-correct answer can be produced by an invalid trace: the query happened to return the right rows for reasons unrelated to the reasoning that was supposed to get there. Answer-only evaluation counts that as a success.
The underlying failure mode has a name here — the Structure Gap. Natural-language reasoning and free-form rationales do not reliably specify the operator-level programs that real systems need. A deployed agent has to hand something executable to a database. A paragraph of plausible reasoning is not that, and the gap between the two is where incorrect-but-correct-looking outputs come from.
This is not a critique of benchmark design in the abstract; it is a claim about deployment. A system certified at some accuracy can carry a substantial rate of answers that no real system should act on, and nothing in the evaluation distinguishes them.
THE STRUCTURE GAP
what the model produces: natural-language reasoning,
free-form rationale
what a real system needs: an operator-level program
(executable, checkable)
|
v
the rationale does not reliably SPECIFY the program
-> a correct-looking answer can rest on an invalid computation
|
v
answer-only evaluation:
"output matches the reference" -> SUCCESS
|
v
[GAP] in structured-data tasks, a benchmark-correct
answer can be produced by an invalid trace
The Increment
One sentence: Before this paper, a data agent was certified by matching the reference answer; after it, Trace Integrity names seven properties the computation behind an answer must have, and a measured CAIT rate shows that 45.8-59.1% of correct answers were resting on invalid traces.
Core Mechanism
Trace Integrity is defined as a deployment reliability criterion: the computation recorded behind an answer must be explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. Seven properties, and they are worth reading as a checklist rather than a definition. Each rules out a distinct way an answer can be right for the wrong reason — a trace that is implicit cannot be reviewed, one that is not executable cannot be run, one that is not operator-faithful got the answer by a route the system did not describe, and so on.
Two constructs make it operational.
Execution contracts are the structured artifacts that bind user intent to schema elements, operator plans, assumptions, executable queries, verification status, and final-answer linkage. They are the answer to the Structure Gap: instead of hoping a rationale specifies a program, the contract is the specification, and everything the deployment needs to check is attached to it.
CAIT rate — Correct Answer / Invalid Trace — measures how often answer-only evaluation counts computationally unsupported outputs as successes. This is the metric that makes the problem legible, because it is expressed as a failure rate of the existing evaluation practice rather than as a property of any particular system.
The empirical demonstration runs three SQL pipelines on BIRD Mini-Dev, and the three signals come apart cleanly:
PIPELINE Direct SQL Op Summary + SQL Contract-First SQL
answer accuracy 20% 22% 24%
Trace Integrity Pass Rate 39% 43% 40%
CAIT rate 55% 59.1% 45.8%
Read the first row against the third. The pipeline with the highest answer accuracy (24%) has the lowest CAIT rate (45.8%) — which is directionally reassuring, but note the magnitude: nearly half of its correct answers still rest on traces that fail the integrity criterion. And the pipeline with the second-highest accuracy has the highest CAIT rate at 59.1%. Answer accuracy, trace validity, and silent-failure risk are three distinct evaluation signals, and a single accuracy number reports none of them separately.
TRACE INTEGRITY: SEVEN REQUIRED PROPERTIES
explicit | executable | schema-valid
operator-faithful | replayable | answer-consistent | auditable
|
v
EXECUTION CONTRACT binds:
user intent <-> schema elements
operator plans, assumptions
executable queries, verification status
final-answer linkage
|
v
CAIT = Correct Answer / Invalid Trace rate
= how often answer-only evaluation scores an
unsupported computation as a success
|
v
MEASURED (BIRD Mini-Dev, 3 SQL pipelines)
accuracy 20/22/24 vs pass rate 39/43/40 vs CAIT 55/59.1/45.8
-> three signals, not one
Think of it as a maths exam where only the final answer is marked. A student who gets the right number by a wrong method scores identically to one who did the derivation properly, and if the numbers are large enough, some will arrive at the right answer by coincidence, by pattern-matching previous problems, or by a shortcut that happens to work here and nowhere else. The exam certifies a skill that was not demonstrated. Marking the working as well closes most of it — and note the direction of the paper’s prescription: it does not ask for a better final answer, or a longer rationale. It asks for the working to be executable, replayable and auditable, which are properties of the working itself, not of the number.
Key Concepts
- Trace Integrity: the seven-property criterion for whether the computation behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent and auditable. Its function is to convert “how did you get this” from a narrative question into a checkable one.
- The Structure Gap: natural-language reasoning not reliably specifying operator-level programs. It is the reason Trace Integrity is necessary rather than merely desirable — the deliverable a real system consumes is a program, and a rationale is not one.
- CAIT rate: correct answers resting on invalid traces, expressed as a rate. Its value is that it measures the failure of the evaluation practice, so it is comparable across systems and quantifies how much is currently invisible.
- Execution contracts: structured artifacts binding intent, schema, operator plans, assumptions, executable queries, verification status and answer linkage. They are the constructive answer to the Structure Gap, and their content shows what “auditable” concretely requires.
Framework Shift
Before (answer-only evaluation):
agent produces a rationale + a query
score: does the output match the reference?
-> a coincidentally-correct output scores as a success
-> silent failure risk is unmeasured
After (trace-level evaluation):
agent produces reasoning + an EXECUTION CONTRACT
score: answer accuracy AND trace integrity AND CAIT
-> 24% accuracy can coexist with 45.8% of those
correct answers resting on invalid traces
-> deployment risk is reported separately
From certifying that the answer matched, to certifying that the computation behind it is explicit and replayable, the core shift is that in structured-data tasks the artifact a system consumes is a program, so the program is what must be validated.
Expert Assessment
Problem choice: Very good, and correctly scoped to structured-data deployment. The claim that answer accuracy is an insufficient reliability signal is easy to assert and usually asserted loosely; identifying the Structure Gap as the mechanism, and giving the criterion seven checkable properties, is what makes it a contribution rather than a complaint.
Method maturity: The design is an evaluation framework rather than a model, and execution contracts are the substantive part — they specify what an auditable trace contains, which is more useful than a definition of auditability. The CAIT rate is well chosen as the headline because it measures the existing practice’s blind spot rather than proposing a new score nobody has a baseline for.
Experimental integrity: The demonstration is small and the paper is upfront about its nature — three pipelines on one benchmark, presented as showing that the three signals separate rather than as a definitive measurement of any system. Reporting 20-24% accuracy alongside 39-43% pass rates is honest in both directions: the accuracy is low enough that the pipelines are not strong, and the pass rates are low enough that the framework is not a rubber stamp. The obvious limitation is scale, and a larger study across more agent designs would establish how the CAIT rate distributes in practice.
Writing quality: The seven properties read as a checklist, which is the right form for a deployment criterion, and the CAIT definition is stated precisely enough to be implemented. The paper would be more actionable with one concrete execution contract shown in full — a single query, its operator plan, its assumptions and its verification status — since that is the artifact teams would have to start producing.
Verdict: accept — it names a deployment failure mode that answer-only evaluation structurally cannot see, and supplies the criterion and the metric needed to measure it.
Takeaways
- In structured-data tasks, evaluate the computation, not just the output. A correct answer can rest on a trace no production system should accept.
- Track a CAIT-style rate in production. It measures how much of your apparent accuracy is unsupported, which is the number that predicts silent failures.
- Make the trace executable and replayable rather than merely narrated. Auditability is a property of the artifact, so a prose rationale cannot be audited no matter how detailed it is.
- Read accuracy and integrity as separate columns. Here a pipeline with higher accuracy also carried a higher rate of invalid traces, so one number cannot substitute for the other.
论文: 2608.26036 作者: Srimonti Dutta, Akshata Kishore Moharir 分类: cs.AI, cs.CL
缺口
答案准确率被当作大模型数据智能体的可靠性信号，而对结构化数据任务来说，它是一个错误的信号。 一个在基准上正确的答案，可以建立在一条无效轨迹之上：查询恰好返回了正确的行，而其理由与”本该把它推导出来的那套推理”毫无关系。只看答案的评测会把它算作成功。
底层的失效模式在文中有一个名字——结构缺口（Structure Gap）。 自然语言推理与自由形式的解释，并不能可靠地指定真实系统所需要的算子级程序。被部署的智能体必须把某种可执行的东西交给数据库。一段看起来合理的推理文字并不是那个东西，而两者之间的落差，正是”错得看起来却对”的输出来源。
这不是对基准设计的抽象批评，而是关于部署的主张。 一个在某个准确率上取得认证的系统，可能携带着相当比例的、任何真实系统都不该据以行动的答案——而评测里没有任何东西能把它们区分出来。
结构缺口
模型产出的： 自然语言推理、自由形式解释
真实系统需要的：算子级程序（可执行、可校验）
|
v
解释并不能可靠地「指定」那个程序
-> 一个看起来正确的答案，可以建立在无效计算上
|
v
只看答案的评测：
"输出与参考答案一致" -> 成功
|
v
[缺口] 在结构化数据任务中，一个基准上正确的答案
可以出自一条无效轨迹
增量
一句话: 在这篇论文之前，数据智能体是靠”与参考答案一致”取得认证的；在这篇论文之后，轨迹完整性（Trace Integrity）点名了一个答案背后的计算必须具备的七项性质，而实测的 CAIT 率表明：45.8%~59.1% 的正确答案建立在无效轨迹上。
核心机制
轨迹完整性被定义为一个部署可靠性判据：一个答案背后被记录下来的计算，必须是显式的、可执行的、符合 schema 的、算子忠实的、可重放的、与答案一致的、以及可审计的。七项性质，值得当成一份检查清单来读，而不是一个定义。每一项都排除了一种”答案对、理由错”的具体方式——不显式的轨迹无法被复核，不可执行的轨迹无法被运行，不忠实于算子的轨迹是以系统未曾描述的另一条路径拿到答案的，如此等等。
两个构造让它变得可操作。
执行契约（execution contracts）是把用户意图与 schema 元素、算子计划、假设、可执行查询、验证状态、以及最终答案关联绑定在一起的结构化产物。它们就是”结构缺口”的答案：与其指望一段解释能指定一个程序，契约本身就是那份规格说明，部署所需校验的一切都挂在上面。
CAIT 率——Correct Answer / Invalid Trace——衡量的是**“只看答案”的评测有多少次把计算上站不住的输出算成了成功**。正是这个指标让问题变得可读，因为它是用既有评测实践的失败率来表达的，而不是用某个特定系统的属性。
实证演示在 BIRD Mini-Dev 上跑了三条 SQL 流水线，而三个信号干净地分了家：
流水线 Direct SQL 操作摘要+SQL 契约优先 SQL
答案准确率 20% 22% 24%
轨迹完整性通过率 39% 43% 40%
CAIT 率 55% 59.1% 45.8%
把第一行与第三行对着读。 答案准确率最高（24%）的那条流水线，CAIT 率最低（45.8%）——方向上令人安心；但请注意量级：它将近一半的正确答案，依然建立在通不过完整性判据的轨迹上。而准确率排第二的那条，CAIT 率反而是最高的 59.1%。 答案准确率、轨迹有效性、以及静默失效风险，是三个彼此不同的评测信号，而一个准确率数字并不分别报告其中任何一个。
轨迹完整性：七项必需性质
显式 | 可执行 | 符合 schema
算子忠实 | 可重放 | 与答案一致 | 可审计
|
v
执行契约绑定：
用户意图 <-> schema 元素
算子计划、假设
可执行查询、验证状态
与最终答案的关联
|
v
CAIT = 正确答案 / 无效轨迹 的比率
= 「只看答案」的评测把站不住的计算
算作成功的频率
|
v
实测（BIRD Mini-Dev，三条 SQL 流水线）
准确率 20/22/24 vs 通过率 39/43/40 vs CAIT 55/59.1/45.8
-> 三个信号，而不是一个
可以用**“只给最终答案打分的数学考试”来理解这件事： 一个用错方法却恰好算出正确数字的学生，与一个老老实实推导的学生拿到同样的分数；而只要数字足够大，总会有人靠巧合、靠套用以前见过的题型、或靠一个”只在这里管用、别处都不管用”的捷径得到正确答案。 这场考试认证的是一项从未被展示过的技能**。 把过程也纳入评分就能堵住大部分——而请注意本文处方的方向：它并不要求更好的最终答案，也不要求更长的解释。它要求过程本身可执行、可重放、可审计；这些是过程的属性，而不是那个数字的属性。
关键概念
- 轨迹完整性（Trace Integrity）: 判断一个答案背后的计算是否”显式、可执行、符合 schema、算子忠实、可重放、与答案一致、可审计”的七项判据。它的作用是把”你是怎么得到这个的”从一个叙事性问题，变成一个可校验的问题。
- 结构缺口（Structure Gap）: 自然语言推理无法可靠地指定算子级程序。它是”轨迹完整性之所以必要、而不仅仅是有益”的原因——真实系统所消费的交付物是一个程序，而一段解释不是程序。
- CAIT 率: 建立在无效轨迹之上的正确答案所占的比率。它的价值在于度量的是评测实践的失效，因此可以在不同系统之间比较，并把”当前有多少东西是不可见的”量化出来。
- 执行契约: 把意图、schema、算子计划、假设、可执行查询、验证状态与答案关联绑定在一起的结构化产物。它们是结构缺口在建设性方向上的答案；而它们的内容，正好展示了”可审计”具体要求什么。
框架转变
之前（只看答案的评测）：
智能体产出：一段解释 + 一条查询
评分：输出是否与参考答案一致？
-> 靠巧合正确的输出被记为成功
-> 静默失效风险完全未被度量
之后（轨迹层面的评测）：
智能体产出：推理 + 一份「执行契约」
评分：答案准确率 AND 轨迹完整性 AND CAIT
-> 24% 的准确率可以与「其中 45.8% 的正确答案
建立在无效轨迹上」并存
-> 部署风险被单独报告
从”认证答案是否一致”，转变为”认证答案背后的计算是否显式并可重放”，核心转变在于：在结构化数据任务中，系统消费的产物是一个程序，因此必须被校验的就是那个程序。
专家评审
选题眼光: 很好，而且范围正确地限定在结构化数据部署上。 “答案准确率不是一个充分的可靠性信号”这句话很容易说，也常常被说得很空泛；而把”结构缺口”点名为机制，并给这条判据七项可校验的性质，才使它成为一项贡献，而不只是一句抱怨。
方法成熟度: 它是一套评测框架而不是一个模型，而实质部分在于执行契约——它们明确了”一条可审计的轨迹里包含什么”，这比给”可审计性”下一个定义更有用。 CAIT 率作为头条选得好，因为它度量的是既有实践的盲区，而不是提出一个”没有任何基线可参照”的新分数。
实验诚意: 演示规模不大，而论文对此也如实说明——三条流水线、一个基准，定位是”展示三个信号确实分家”，而不是”对某个系统给出定论式测量”。 同时报出 20%~24% 的准确率与 39%~43% 的通过率，在两个方向上都是诚实的：准确率低到说明这些流水线并不强，通过率也低到说明这套框架不是橡皮图章。显而易见的局限是规模；一项覆盖更多智能体设计的研究，才能确立 CAIT 率在实践中的分布形态。
写作功力: 七项性质读起来像一份检查清单，而这对一条部署判据来说是正确形态；CAIT 的定义也精确到足以被实现。 若能完整展示一份具体的执行契约——一条查询、它的算子计划、它的假设、它的验证状态——论文会更可操作，因为那正是各团队必须开始产出的东西。
判决: 接收（Accept） — 它点名了一种”只看答案”的评测在结构上无法看到的部署失效模式，并给出了度量它所需的判据与指标。
要点总结
- 在结构化数据任务中，评测计算过程，而不只是输出。一个正确答案可以建立在任何生产系统都不该接受的轨迹之上。
- 在生产中跟踪一个 CAIT 式的比率。它度量的是”你的表观准确率中有多少是站不住的”，而那才是预测静默失效的数字。
- 让轨迹可执行、可重放，而不只是被叙述。可审计性是产物的属性，因此一段文字解释无论多详细都无法被审计。
- 把准确率与完整性当成两列来读。在这里，准确率更高的一条流水线同时携带了更高的无效轨迹率——一个数字无法替代另一个。