Paper: 2609.10494 Authors: Blake Stenstrom, Charangan Vasantharajan, Brian Sathianathan Categories: cs.AI, cs.CL, cs.LG
The Gap
The opening distinction is the whole paper in one line: enterprises deploy systems, not checkpoints. What a deployment can actually do depends jointly on weights, serving route, precision, output contract, and harness — five things, only the first of which a model card describes.
And yet: all 18 audited benchmarks score advertised model identifiers. So the field’s instrument measures one component and reports it as the capability of the whole. The paper calls this measurement error and, importantly, does not stop at the complaint — the contribution is a protocol that makes the error reportable. The framing is what makes the work constructive: the problem is not that benchmarks exist, it is that a reportable quantity is being reported as a different quantity.
THE DISTINCTION IN ONE LINE
ENTERPRISES DEPLOY SYSTEMS, NOT CHECKPOINTS
|
v
what a deployment can DO depends JOINTLY on:
WEIGHTS | SERVING ROUTE | PRECISION | OUTPUT CONTRACT | HARNESS
<- five things, only the FIRST of which a MODEL CARD describes
|
v
AND YET: ALL 18 AUDITED BENCHMARKS SCORE ADVERTISED MODEL IDENTIFIERS
-> the field's instrument measures ONE COMPONENT and reports it
as the capability of the WHOLE
|
v
[THE PAPER CALLS THIS MEASUREMENT ERROR]
and does NOT stop at the complaint: the contribution is
A PROTOCOL THAT MAKES THE ERROR REPORTABLE
<- the problem is not that benchmarks exist; it is that a
REPORTABLE QUANTITY is being reported AS A DIFFERENT QUANTITY
The Increment
One sentence: Before this paper, enterprise capability was measured by the model identifier; after it, a three-part protocol binds a route to the evaluation contract, keeps failure in the score, and adjudicates score-blindly, with the reference instantiation deliberately sealed.
Core Mechanism
The protocol has three parts, and each addresses a specific way the measurement can go wrong.
A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it. Two words carry it. Gold-blind means the preflight has no access to the expected answers, so it cannot be tuned to pass. And it runs before any task reaches the route, which is what separates “this route can’t do the task format” from “this route does the task badly” — without the preflight, a route that cannot produce the required output shape would simply score near zero and be reported as weak rather than unsupported.
A reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out. That pairing is carefully balanced: a failed response counts against the score (so a system is not rewarded for refusing to answer), while a route that failed the preflight is excluded rather than scored (so an unsupported configuration is not reported as an incapable one). Naively, one does the opposite of the other; getting both requires the preflight to exist first, which is why the three parts are ordered.
Adjudication is structurally score-blind. Whoever decides outcomes does not see the scores. It is a governance property rather than a statistical one, and it addresses the failure where a judge rationalises a result it has already seen.
The reference instantiation is sealed on purpose: 128 locked tasks and 987 assertions over document, spreadsheet, chart, tool and database work, stays sealed: the procedure is the artifact, not the corpus. That sentence is a design position, and it is defensible — publishing the corpus would let routes be tuned against it, converting a measurement protocol back into a leaderboard. The cost is that others cannot verify the specific numbers, which the paper accepts in favour of distributing the procedure, its algorithms, classification tables, request contract and manifest schemas.
Four results across eleven systems, each of which is a specific defect in identifier-based measurement:
Capability availability is measurable, and the identifier hides it. Two complete single-route runs on identical weights later failed distinct predicates of the finalized binding gate, while a third passed that gate before a fresh run. The advertised identifier exposed neither limit. So the same advertised model can be unavailable through one route and available through another — invisible to any measurement that reads the name.
Discrimination is not uniform. Four of seven suites saturate under a six-system band, with the spread coming almost entirely from governed database work and multi-tab joins. And the consequence is a reporting change: interval-backed resolution groups, not ranks — because ranking systems the suites cannot separate produces a false order. Reinforcing that, two of the nominal five-label output’s four cuts fail multiplicity adjustment, so the label scheme itself overstates how finely it discriminates.
Serving-arm choice moved a declared revision and precision from 77.38 to 82.54, paired interval [0.11, 10.60] — a real difference from an arm choice, not from a model change. The honesty here is in the caveats: the arms differ in access mode, harness generation, and the serving tool-call parser, and harness generation is a property of the evaluator, not any endpoint. So part of that movement is the instrument, and the paper says so rather than attributing all of it to the serving config.
Excluding failed responses from denominators changes the point ordering — therefore reliability inclusion changes a conclusion, not its wording. That is the sharpest of the four: a seemingly technical scoring decision determines which system appears better.
THE PROTOCOL HAS THREE PARTS, each addressing a specific way the
measurement can go wrong
[1] A GOLD-BLIND CAPABILITY-BINDING PREFLIGHT verifies that a route
can EXECUTE THE EVALUATION CONTRACT BEFORE ANY TASK REACHES IT
"GOLD-BLIND" -> no access to expected answers
-> it CANNOT BE TUNED TO PASS
runs BEFORE ANY TASK REACHES the route
-> separates "this route CANNOT DO THE TASK FORMAT" from
"this route does the task BADLY"
-> without it, a route that cannot produce the required
output shape scores NEAR ZERO and is reported as WEAK
rather than UNSUPPORTED
[2] A RELIABILITY-INCLUSIVE FIRST-PASS SCORING RULE keeps FAILURE IN
THE SCORE while keeping UNSUPPORTED CAPABILITY OUT
<- carefully BALANCED pairing:
a FAILED RESPONSE counts against the score
(so a system is not rewarded for REFUSING TO ANSWER)
a route that FAILED THE PREFLIGHT is EXCLUDED, not scored
(so an UNSUPPORTED config is not reported as INCAPABLE)
<- naively one does the OPPOSITE of the other; getting both
requires the preflight to exist FIRST
-> which is why the three parts ARE ORDERED
[3] ADJUDICATION IS STRUCTURALLY SCORE-BLIND
whoever decides outcomes DOES NOT SEE THE SCORES
<- a GOVERNANCE property, not a statistical one
<- addresses the failure where a judge RATIONALISES a result
it has ALREADY SEEN
THE REFERENCE INSTANTIATION IS SEALED ON PURPOSE
128 locked tasks and 987 assertions over document, spreadsheet,
chart, tool and database work
-> "THE PROCEDURE IS THE ARTIFACT, NOT THE CORPUS"
<- defensible: publishing the corpus would let routes be TUNED
AGAINST IT, converting a measurement protocol back into a
LEADERBOARD
<- the cost is that others cannot verify the specific numbers,
accepted in favour of distributing the PROCEDURE, its
ALGORITHMS, classification tables, request contract and
manifest schemas
FOUR RESULTS ACROSS ELEVEN SYSTEMS -- each a specific defect in
identifier-based measurement
[a] CAPABILITY AVAILABILITY IS MEASURABLE, AND THE IDENTIFIER
HIDES IT
two complete SINGLE-ROUTE runs on IDENTICAL WEIGHTS later
failed DISTINCT predicates of the finalized binding gate,
while a third passed that gate before a fresh run
THE ADVERTISED IDENTIFIER EXPOSED NEITHER LIMIT
-> the same advertised model can be UNAVAILABLE through one
route and available through another, INVISIBLE to any
measurement that READS THE NAME
[b] DISCRIMINATION IS NOT UNIFORM
FOUR OF SEVEN SUITES SATURATE under a SIX-SYSTEM BAND
spread comes ALMOST ENTIRELY from GOVERNED DATABASE WORK and
MULTI-TAB JOINS
-> REPORTING CHANGE: INTERVAL-BACKED RESOLUTION GROUPS,
NOT RANKS
<- ranking systems the suites cannot separate produces a
FALSE ORDER
reinforcing this: TWO of the nominal five-label output's
FOUR CUTS FAIL MULTIPLICITY ADJUSTMENT
-> the label scheme itself OVERSTATES how finely it
discriminates
[c] SERVING-ARM CHOICE MOVED a declared revision and precision from
77.38 to 82.54, paired interval [0.11, 10.60]
<- a REAL DIFFERENCE from an ARM CHOICE, not a model change
HONESTY IN THE CAVEATS: the arms differ in ACCESS MODE,
HARNESS GENERATION, and the SERVING TOOL-CALL PARSER
and HARNESS GENERATION IS A PROPERTY OF THE EVALUATOR, NOT
ANY ENDPOINT
-> part of that movement is THE INSTRUMENT, and the paper
says so rather than attributing all of it to the config
[d] EXCLUDING FAILED RESPONSES FROM DENOMINATORS CHANGES THE POINT
ORDERING
-> RELIABILITY INCLUSION CHANGES A CONCLUSION, NOT ITS WORDING
<- the SHARPEST of the four: a seemingly TECHNICAL SCORING
DECISION determines WHICH SYSTEM APPEARS BETTER
Think of it as inspecting a restaurant’s kitchen rather than its signboard. The signboard names the chef, and a customer reasonably treats the name as the answer. But what arrives at the table depends on which kitchen the order was routed to, what equipment it has, and who is actually cooking — and two branches under the same name can differ. The three protocol parts map cleanly: check before ordering whether this kitchen can cook the dish at all (otherwise a branch with no oven gets recorded as a bad cook rather than an unqualified one); count a burnt dish against the kitchen but exclude the branch that lacks the equipment; and have someone other than the person who saw the scores decide what the outcome means. And sealing the test menu, rather than publishing it, is the choice not to let every kitchen start cooking only the dishes on the exam.
Key Concepts
- Systems versus identifiers: capability depends on weights, route, precision, output contract and harness, while benchmarks score the name. It is the reframing that turns a complaint into a measurement problem.
- Gold-blind preflight before the task: verifying the route can execute the contract, without access to expected answers. It separates unsupported from incapable, which are reported identically otherwise.
- Reliability-inclusive scoring with preflight exclusion: failure counts, but unsupported configurations do not. The pairing is only coherent because the preflight comes first.
- Score-blind adjudication: a governance property, not a statistical one, aimed at the judge who has already seen the number.
- Sealing the corpus: the procedure is released and the tasks are not, so routes cannot be tuned against the exam. It trades external verification of specific numbers for the protocol’s integrity.
- Resolution groups instead of ranks: interval-backed groupings, because four of seven suites cannot separate the systems being ranked and two label cuts fail multiplicity adjustment.
Framework Shift
Before (the identifier as the measurement):
report the advertised model name
-> one of five determinants of usable capability
-> a route that cannot execute the contract scores near zero and is
reported as weak
-> ranking systems the suites cannot separate produces a false order
After (the route and the contract as the measurement):
gold-blind preflight binds the route to the contract first
reliability-inclusive scoring keeps failure in and unsupported out
adjudication is score-blind
-> same weights, different routes: distinct gate failures, invisible
to the identifier
-> resolution groups, not ranks, where discrimination saturates
-> the corpus stays sealed; the procedure is released
From measuring the name on the package, to measuring the route and verifying it can execute the contract before scoring it, the core shift is that usable capability is a property of a deployed system and the instrument has to be able to say so.
Expert Assessment
Problem choice: Excellent, and it names a measurement error the field has been living with rather than a benchmark shortfall. “Enterprises deploy systems, not checkpoints” is the kind of sentence that reorganises a literature, and the decision to build a protocol rather than to document the gap is what makes the contribution usable.
Method maturity: The three parts are ordered by necessity, not convention: reliability-inclusive scoring is only coherent once the gold-blind preflight exists, since otherwise there is no way to distinguish incapable from unsupported. That is a genuine design argument. The four results are each tied to a distinct defect, and the fourth — that a scoring convention changes the ordering — is the most consequential because it is the least visible. Sealing the reference instantiation is a defensible position given the stated reason, and releasing the algorithms, tables, contract and schemas keeps the protocol reproducible in the sense that matters.
Experimental integrity: The paper is unusually candid about the limits of its own instrument: it reports that arms differ in ways including a property of the evaluator, and therefore declines to attribute the 77.38-to-82.54 movement entirely to the serving configuration. Reporting that two label cuts fail multiplicity adjustment is self-critical in the same spirit. The limitation follows from the sealing decision: because the corpus is not released, the specific numbers cannot be independently verified, so the paper’s claims rest on the protocol being correctly applied — which is exactly the role the released algorithms and manifest schemas are meant to serve, and a reader must accept the trade.
Writing quality: The four results are the densest part and are enumerated in a way that survives being read quickly, which suits a measurement paper whose audience includes practitioners rather than only researchers. Because the seal is the most contestable choice, a short passage on how a third party could certify their own instantiation — using the released contract and schemas — would pre-empt the obvious objection.
Verdict: strong accept — it treats the gap between deployed capability and advertised identifiers as a reportable measurement error, supplies an ordered protocol whose parts depend on each other for coherence, and reports findings that undercut its own instrument along with the rest.
Takeaways
- Check what your instrument actually measures. If capability depends on five things and you score one, the report describes a different object than the reader assumes.
- Verify support before scoring. Without a preflight, a configuration that cannot execute the contract is reported as weak rather than unsupported.
- Decide whether failure counts before you compare systems. A scoring convention about denominators changed the ordering here, so it is a conclusion, not a formatting choice.
- Report groupings when the suites cannot separate. Ranks over indistinguishable systems are a false order, and interval-backed groups are the honest form.
论文: 2609.10494 作者: Blake Stenstrom, Charangan Vasantharajan, Brian Sathianathan 分类: cs.AI, cs.CL, cs.LG
缺口
开篇那个区分就是整篇论文：企业部署的是「系统」，而不是「检查点」。 一个部署实际能做什么，同时取决于权重、服务路由、精度、输出契约与脚手架——五样东西，而模型卡只描述其中的第一样。
然而：受审计的 18 个基准全都在给「广告上的模型标识」打分。 于是这个领域的仪器测的是一个组件，却把它当作整体的能力来报告。论文把这件事叫作测量误差，而且重要的是——它没有停在抱怨上：贡献是**「一套让这个误差可被报告的协议」。** 正是这个框定让工作变得建设性：问题不是”基准不该存在”，而是一个可报告的量正在被当成另一个量来报告。
一句话里的区分
「企业部署的是系统，而不是检查点」
|
v
一个部署「实际能做什么」同时取决于：
「权重 | 服务路由 | 精度 | 输出契约 | 脚手架」
<- 五样东西，而「模型卡只描述第一样」
|
v
然而：受审计的 18 个基准「全都在给广告上的模型标识打分」
-> 这个领域的仪器测的是「一个组件」，
却把它当作「整体」的能力来报告
|
v
[论文称之为「测量误差」]
而且没有停在抱怨上：贡献是
「一套让这个误差可被报告的协议」
<- 问题不是"基准不该存在"，而是
"一个「可报告的量」正在被当成「另一个量」来报告"
增量
一句话: 在这篇论文之前，企业能力是靠模型标识来测量的；在这篇论文之后，一套三部分协议把路由与评测契约绑定、把失败保留在分数里、并做”对分数盲”的裁决，而参考实现被刻意封存。
核心机制
协议有三部分，而每一部分都针对”测量可能出错的某一种具体方式”。
一个「对金标准盲」的能力绑定预检，在任何任务抵达之前先验证该路由能否执行评测契约。 有两个词在承重。“对金标准盲”意味着预检无法访问预期答案，因此它无法被调到通过。而它在任何任务抵达该路由之前运行，这才把”这条路由做不了这种任务格式”与”这条路由把任务做得很差”分开——没有预检，一条无法产出所需输出形态的路由会拿到接近零的分数，并被报告为弱，而不是不受支持。
一条「把可靠性纳入」的首轮评分规则，把失败保留在分数里，同时把不受支持的能力排除在外。 这个配对被小心地平衡过：一个失败的回答要计入分数（这样系统不会因为”拒绝作答”而获得好处），而一条未通过预检的路由被排除、而不是被打分（这样一个不受支持的配置不会被报告成没有能力的）。朴素地做，这两者会互相矛盾；要同时做到，必须先有预检——这正是三部分有序的原因。
裁决在结构上「对分数盲」。 决定结果的人看不到分数。这是一个治理属性、而不是统计属性；它针对的是”评判者在已经看到结果之后为它找理由”这种失效。
参考实现是被刻意封存的：128 个锁定任务、987 条断言，覆盖文档、表格、图表、工具与数据库工作；它保持封存：交付物是「流程」，而不是「题目集」。 这句话是一个设计立场，而且是站得住的——公开题目集会让各路由针对它调优，把一个测量协议又变回排行榜。代价是别人无法验证那些具体数字；论文接受这个代价，以换取分发流程、其算法、分类表、请求契约与清单模式。
跨 11 个系统的四项结果，每一项都是”基于标识的测量”的一个具体缺陷：
能力的可用性是可测的，而标识把它藏了起来。 两个在相同权重上完整跑通的单路由运行，后来在最终绑定闸门上分别失败了不同的谓词；而第三条路由在另一次全新运行之前通过了那道闸门。广告上的标识「既没有暴露」其中任何一个限制。 也就是说，同一个广告模型，可能在一条路由上不可用、在另一条上可用——而这对任何”读名字”的测量都是不可见的。
区分度并不均匀。 七个套件中有四个在「六系统区间」内饱和，而差距几乎全部来自受治理的数据库工作与多标签页连接。后果是一项报告方式的改变：用「区间支撑的分辨组」，而不是「名次」——因为给套件根本区分不开的系统排名，会产生一个虚假的顺序。与此呼应：名义上五标签输出中，四个切点里有两个通不过多重性校正，也就是说这个标签方案本身夸大了它的区分精度。
服务臂的选择把一个已声明的修订与精度从 77.38 移动到 82.54，配对区间 [0.11, 10.60]——这是一个来自臂的选择的真实差异，而不是来自模型更换。诚实之处在保留条件里：这些臂在访问方式、脚手架生成、以及服务端的工具调用解析器上不同；而且脚手架生成是我们评测器自身的属性，不是任何端点的属性。也就是说，这个位移中有一部分是仪器，论文如实说出来，而没有把它全部归给服务配置。
把失败回答从分母中排除，会改变点排序——因此**“把可靠性纳入”改变的是一个结论，而不只是它的措辞。** 这是四项中最锋利的一条：一个看似技术性的评分决定，决定了哪个系统看起来更好。
协议有三部分，每一部分都针对"测量可能出错的某一种具体方式"
[1] 一个「对金标准盲」的能力绑定预检，在任何任务抵达之前
验证该路由能否「执行评测契约」
"对金标准盲" -> 无法访问预期答案
-> 它「无法被调到通过」
在任何任务抵达之前运行
-> 把"这条路由「做不了」这种任务格式"与
"这条路由把任务做得「很差」"分开
-> 没有它，一条无法产出所需输出形态的路由会拿到
接近零的分数，并被报告为「弱」而不是「不受支持」
[2] 一条「把可靠性纳入」的首轮评分规则，
把「失败保留在分数里」，同时把「不受支持的能力排除在外」
<- 被小心平衡过的配对：
失败的回答「计入分数」
（系统不会因为"拒绝作答"而获得好处）
未通过预检的路由被「排除」，而不是被打分
（不受支持的配置不会被报告成"没有能力的"）
<- 朴素地做，这两者会互相「矛盾」；要同时做到，
必须先有预检
-> 这正是三部分「有序」的原因
[3] 裁决在结构上「对分数盲」
决定结果的人「看不到分数」
<- 这是「治理」属性，而不是统计属性
<- 针对"评判者在已经看到结果之后为它找理由"这种失效
「参考实现被刻意封存」
128 个锁定任务、987 条断言，覆盖文档、表格、图表、工具与
数据库工作
-> "交付物是「流程」，而不是「题目集」"
<- 站得住：公开题目集会让各路由「针对它调优」，
把一个测量协议又变回「排行榜」
<- 代价是别人无法验证那些「具体数字」，
论文接受这个代价，以换取分发「流程」、其「算法」、
分类表、请求契约与清单模式
「跨 11 个系统的四项结果」——每一项都是"基于标识的测量"
的一个具体缺陷
[a] 能力的可用性可测，「而标识把它藏了起来」
两个在「相同权重」上完整跑通的「单路由」运行，后来在
最终绑定闸门上分别失败了「不同的谓词」；而第三条路由
在另一次全新运行之前通过了那道闸门
「广告上的标识既没有暴露其中任何一个限制」
-> 同一个广告模型可能在一条路由上不可用、在另一条上
可用，而对任何"读名字"的测量「不可见」
[b] 区分度并不均匀
「七个套件中有四个」在「六系统区间」内饱和
差距「几乎全部来自受治理的数据库工作与多标签页连接」
-> 「报告方式的改变」：用「区间支撑的分辨组」，
而不是「名次」
<- 给套件根本区分不开的系统排名，会产生「虚假的顺序」
与此呼应：名义上五标签输出中，「四个切点里有两个
通不过多重性校正」
-> 这个标签方案本身「夸大了」它的区分精度
[c] 服务臂的选择把一个已声明的修订与精度从 77.38 移动到 82.54，
配对区间 [0.11, 10.60]
<- 一个来自「臂的选择」的「真实差异」，不是模型更换
诚实在保留条件里：这些臂在「访问方式、脚手架生成、
服务端工具调用解析器」上不同
而「脚手架生成是我们评测器自身的属性，不是任何端点的属性」
-> 这个位移中有一部分是「仪器」，论文如实说出来，
而没有把它全部归给服务配置
[d] 「把失败回答从分母中排除，会改变点排序」
-> "把可靠性纳入"改变的是一个「结论」，而不只是它的措辞
<- 四项中最锋利的一条：一个看似「技术性的评分决定」，
决定了「哪个系统看起来更好」
可以用**“检查一家餐厅的后厨，而不是看它的招牌”来理解这件事： 招牌上写着主厨的名字，顾客把那个名字当成答案，是合理的。但端上桌的东西取决于订单被路由到哪个后厨、那里有什么设备、以及究竟是谁在做菜**——而同一个名字下的两家分店可以差别很大。 协议的三部分对得很整齐：在下单之前先确认这个后厨究竟能不能做这道菜（否则一家没有烤箱的分店会被记成”厨子不行”，而不是”没这个条件”）；烧糊的菜要算在这家后厨头上，但缺乏设备的那家分店要被排除；并且由看到分数之外的人来决定结果意味着什么。 而把测试菜单封存、而不是公开，就是那个”不让每家后厨都开始只做考卷上那几道菜”的选择。
关键概念
- 系统 vs 标识: 能力取决于权重、路由、精度、输出契约与脚手架，而基准给”名字”打分。正是这个重构把抱怨变成测量问题。
- 任务之前的对金标准盲预检: 在不访问预期答案的前提下验证路由能执行契约。它把”不受支持”与”没有能力”分开——而这两者本来会被报成同一件事。
- 把可靠性纳入、并用预检做排除: 失败计入，但不受支持的配置不计。这个配对只有在预检先存在时才自洽。
- 对分数盲的裁决: 一个治理属性、而非统计属性，针对的是”已经看到数字的评判者”。
- 封存题目集: 发布流程、不发布任务，使各路由无法针对考卷调优。它用”无法外部验证具体数字”换取协议的完整性。
- 分辨组而非名次: 用区间支撑的分组，因为七个套件中有四个分不开被排名的那些系统，而两个标签切点通不过多重性校正。
框架转变
之前（以标识作为测量）：
报告广告上的模型名
-> 它只是"可用能力"五个决定因素之一
-> 无法执行契约的路由拿到接近零的分数，被报告为"弱"
-> 给套件分不开的系统排名，产生虚假顺序
之后（以路由与契约作为测量）：
对金标准盲的预检先把路由与契约绑定
把可靠性纳入的评分：失败计入、不受支持的排除
裁决对分数盲
-> 相同权重、不同路由：不同的闸门失败，对标识不可见
-> 区分度饱和处用分辨组，而不是名次
-> 题目集保持封存；流程被发布
从”测量包装上的名字”，转变为”测量路由、并先验证它能执行契约、再给它打分”，核心转变在于：可用能力是一个「被部署系统」的属性，而仪器必须能说出这件事。
专家评审
选题眼光: 极好，而且它点出的是一个这个领域一直将就着与之共存的测量误差，而不是”基准的不足”。 “企业部署的是系统，而不是检查点”正是那种会重组一批文献的句子；而”造一套协议、而不是记录这个缺口”的决定，才让贡献变得可用。
方法成熟度: 三部分是因必要性而有序，而不是因惯例：“把可靠性纳入的评分”只有在”对金标准盲的预检”存在之后才自洽，否则就无法区分”没有能力”与”不受支持”。这是一个真实的设计论证。 四项结果各对应一个不同的缺陷，而第四项——一个评分惯例改变了排序——最有后果，因为它最不可见。 封存参考实现，在给出所述理由的前提下是一个站得住的立场；而发布算法、表、契约与模式，让协议在真正要紧的意义上保持可复现。
实验诚意: 论文对自身仪器的局限异常坦率：它报告各臂在若干方面不同，其中包括评测器自身的属性，因此拒绝把 77.38 到 82.54 的位移全部归给服务配置。报告”两个标签切点通不过多重性校正”，同样体现这种自我批评。 局限来自封存这一决定：由于题目集未公开，具体数字无法被独立验证，因此论文的主张依赖于”协议被正确应用”——而这恰恰是已发布的算法与清单模式所要承担的角色，读者必须接受这笔交换。
写作功力: 四项结果是最密的部分，而它们的列举方式能经受快速阅读——这很适合一篇受众包含实践者的测量论文。 由于”封存”是最可能招致质疑的选择，若能补一小段讲清”第三方如何用已发布的契约与模式来认证自己的实例”，就能先发制人地回应那个显而易见的反对。
判决: 强接收（Strong Accept） — 它把”部署能力”与”广告标识”之间的落差当作可报告的测量误差来处理，给出一套各部分互相依赖才自洽的有序协议，并报告了连同”削弱自身仪器”的发现在内的全部结果。
要点总结
- 检查你的仪器实际测的是什么。如果能力取决于五样东西而你只给一样打分，那么报告描述的对象与读者以为的不是同一个。
- 先验证支持，再打分。没有预检，一个无法执行契约的配置会被报告为”弱”，而不是”不受支持”。
- 在比较系统之前先决定失败是否计入。这里关于分母的评分惯例改变了排序——所以它是一个结论问题，而不是格式选择。
- 当套件分不开时，报告分组。对不可区分的系统排名是虚假顺序，而区间支撑的分组是诚实的形态。