Paper: 2609.05405 Authors: Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda Categories: cs.CL
The Gap
Wearable sensing produces exactly the kind of data that should be useful to an AI assistant: continuous monitoring of physiological and behavioural signals, over months, for an individual. Yet existing benchmarks rarely evaluate whether an AI system can reason over a real user’s longitudinal wearable record.
The absence is consequential rather than merely a coverage gap, because the capability being skipped is the one that would make wearables useful. Answering “what is my resting heart rate” is retrieval. Answering “has my sleep changed since my activity dropped” requires reasoning over a user’s own history, and that is the question a user actually has.
WEARABLE SENSING PRODUCES THE RIGHT KIND OF DATA
CONTINUOUS MONITORING of physiological and behavioural
signals, over MONTHS, FOR AN INDIVIDUAL
|
v
[GAP] existing benchmarks RARELY evaluate whether an AI
system can reason over a REAL USER'S LONGITUDINAL
WEARABLE record
|
v
WHY THE ABSENCE IS CONSEQUENTIAL
the capability being skipped is the one that would make
wearables USEFUL:
"what is my resting heart rate?" -> RETRIEVAL
"has my sleep changed since my activity dropped?"
-> requires reasoning over a USER'S OWN HISTORY
<- and THAT is the question a user actually has
The Increment
One sentence: Before this paper, no benchmark tested reasoning over a real user’s longitudinal wearable record; after it, 4,084 questions built from 200 users and up to 500 days each, split by reasoning type, show models ranging from 19.6% to 72.9% and mostly below 60%.
Core Mechanism
The construction choices matter more than the headline number, because the benchmark’s value is in whether it measures the right capability.
Real users and real records: 4,084 ten-option multiple-choice questions built from the wearable time series, blood biomarkers, and demographics of 200 real users, each with up to 500 days of daily measurements. So the questions are grounded in individual histories rather than in population statistics or synthetic trajectories.
Authentic wearable distributions are preserved, including device noise and inter-individual variability. This is the design decision that distinguishes a realistic benchmark from a clean one. Device noise and between-person variation are exactly what make longitudinal reasoning hard in deployment, and a benchmark that smoothed them away would measure a capability nobody deploys against.
Sixteen question types organised along two complementary axes, and the axes are the diagnostic content:
Data versus health reasoning — separating computation over longitudinal measurements from physiological interpretation. These are different competencies and a single score would blend them. A model can aggregate a signal correctly and interpret it wrongly, or the reverse, and only the axis reveals which.
Single- versus cross-signal reasoning — separating reasoning about individual signals from integration of multiple signals. Cross-signal reasoning is where a longitudinal record becomes more than a set of series, and it is plausibly much harder; the axis localises that.
A dual-grounding framework is used to construct questions at scale: literature-grounded physiological findings combined with statistically validated population-grounded physiological patterns. Both are needed and neither is sufficient. Literature grounding ensures the questions reflect real physiology rather than statistical coincidence; population grounding ensures the relationships are actually present in these users’ data rather than assumed from textbooks. The pairing is what allows questions to be generated in volume while remaining meaningful.
The results both validate and delimit the benchmark. Across 14 proprietary and open-source LLMs, performance ranges from 19.6% to 72.9% against a 10% chance baseline. Three things follow:
- The benchmark differentiates model capabilities — a 3.7× spread over a 10-option format means the questions are not guessable and not uniformly beyond reach.
- It is far from solved — the top end is 72.9% on a multiple-choice task with 10 options, so the best model still misses roughly a quarter.
- Most models achieve accuracies below 60% — so the median system is closer to chance than to competence, which is the headline for anyone considering deployment.
CONSTRUCTION: REAL USERS, REAL RECORDS
4,084 TEN-OPTION multiple-choice questions
built from the WEARABLE TIME SERIES, BLOOD BIOMARKERS,
and DEMOGRAPHICS of 200 REAL USERS
each with UP TO 500 DAYS of daily measurements
-> questions grounded in INDIVIDUAL HISTORIES, not
population statistics or synthetic trajectories
AUTHENTIC DISTRIBUTIONS PRESERVED
including DEVICE NOISE and INTER-INDIVIDUAL VARIABILITY
<- the decision that distinguishes a REALISTIC benchmark
from a CLEAN one
<- these are exactly what make longitudinal reasoning hard
in deployment; smoothing them away would measure a
capability nobody deploys against
SIXTEEN QUESTION TYPES ON TWO COMPLEMENTARY AXES
[AXIS 1] DATA vs HEALTH reasoning
computation over longitudinal measurements
vs physiological INTERPRETATION
-> different competencies a single score would blend
-> a model can aggregate correctly and interpret wrongly,
or the reverse; only the axis reveals which
[AXIS 2] SINGLE- vs CROSS-SIGNAL reasoning
individual signals vs INTEGRATION of multiple signals
-> cross-signal is where a record becomes more than a set
of series, and is plausibly much harder
the axis LOCALISES that
DUAL-GROUNDING for scale
LITERATURE-grounded physiological findings
+ STATISTICALLY VALIDATED POPULATION-grounded patterns
<- both needed, neither sufficient:
literature -> questions reflect real PHYSIOLOGY, not
statistical coincidence
population -> the relationships are ACTUALLY PRESENT in
these users' data, not assumed from textbooks
-> allows volume generation that REMAINS MEANINGFUL
RESULTS over 14 proprietary and open-source LLMs
range 19.6% to 72.9% against a 10% CHANCE baseline
- the SPREAD (3.7x) means questions are neither guessable
nor uniformly beyond reach -> it DIFFERENTIATES
- the TOP is 72.9%: still missing roughly a quarter
-> FAR FROM SOLVED
- MOST models below 60%: the median system is closer to
chance than to competence
Think of it as a driving test built from your own commute rather than a standard course. A standard course tells you whether someone can drive. Your commute tells you whether they can drive your route — with the one confusing junction, the school at 3pm, the road that floods. The paper’s insistence on preserving device noise and between-person variability is the same instinct: the interesting failure is not “the model cannot read a heart rate”, it is “the model cannot tell a real trend from a noisy week”. And the axes do the diagnostic work a single score cannot: separating “computed it right but interpreted it wrong” from “could not integrate two signals at all” is the difference between a fixable reasoning gap and a missing capability.
Key Concepts
- Longitudinal reasoning over an individual’s record: the capability the benchmark isolates, and the one that makes wearables useful. It is distinct from retrieval over a signal, which is why the questions are about histories rather than values.
- Preserving authentic distributions: device noise and inter-individual variability retained rather than cleaned. It is what makes the benchmark representative of deployment conditions rather than of idealised data.
- Two diagnostic axes: data versus health reasoning, and single- versus cross-signal. They separate competencies that a single accuracy would blend, which is what turns a score into a diagnosis.
- Dual grounding: literature-grounded findings plus statistically validated population patterns. Literature grounding keeps questions physiological; population grounding keeps them present in the data. Together they permit scale without losing meaning.
- Spread as validity evidence: 19.6% to 72.9% over a 10-option format. A benchmark that everyone scored near chance on, or near ceiling, would not differentiate; this range does.
Framework Shift
Before (benchmarks rarely test longitudinal records):
wearable sensing produces months of physiological data
-> evaluation focuses on signal-level tasks
-> "reason about this user's history" is untested
-> capability gaps in the deployed setting stay invisible
After (a grounded, axis-split benchmark):
4,084 questions from 200 real users, up to 500 days each
authentic noise and variability preserved
16 question types split by reasoning kind
-> models range 19.6%-72.9% against 10% chance; mostly below 60%
-> far from solved on a multiple-choice format
From evaluating models on signal-level tasks while deployment requires history-level reasoning, to a benchmark that requires reasoning over real individual records with real noise, the core shift is that the capability wearables need is longitudinal and conditional on a person, and it had not been measured.
Expert Assessment
Problem choice: Excellent, and it targets a capability that deployment needs and evaluation skipped. Wearables are a natural near-term application for language models precisely because the data is continuous and personal, and the field had been measuring everything except reasoning over that personal continuity.
Method maturity: The two axes are the strongest element, because they convert a score into a diagnosis — a model that aggregates correctly but interprets wrongly is a different problem from one that cannot integrate signals, and a single number cannot say which. Preserving device noise and inter-individual variability is the decision that makes the benchmark honest about deployment. The dual-grounding framework addresses the real difficulty of scaling question generation without drifting into trivia or coincidence, and requiring both literature and population grounding is the right conjunction.
Experimental integrity: The 3.7× spread over a 10-option format is evidence the questions discriminate, and reporting that most models fall below 60% while the best reaches 72.9% is candid in both directions. The scope caveat is that 200 users, while enough for a benchmark, is a modest population for population-grounded claims, so some questions may encode patterns specific to this cohort — and the paper’s own dual-grounding framework is the mitigation, though how much cohort-specificity remains is not quantified.
Writing quality: The axes are named clearly and each is given a one-line rationale, which is what makes the benchmark usable rather than just reportable — you know which axis a failure falls on. Because the practical question is how much of the gap is addressing-able, a short example of one question per axis would make the difficulty concrete for readers who will not read the full item set.
Verdict: strong accept — it builds a realistic and diagnostic benchmark for longitudinal health reasoning over personal wearable records, with axes that localise failures and a grounding framework that permits scale without losing physiological meaning.
Takeaways
- Test reasoning over a record, not retrieval from it. “What is my resting heart rate” and “has my sleep changed since my activity dropped” are different capabilities.
- Preserve the noise. Cleaning device artefacts and inter-individual variability out of a health benchmark removes exactly what makes deployment hard.
- Split the benchmark along competency axes. Data versus health, and single versus cross-signal reasoning, produce diagnoses that a single accuracy blends away.
- Ground generated questions twice. Literature keeps them physiological; population statistics keep them present in the data — and either alone permits a benchmark that measures nothing real.
论文: 2609.05405 作者: Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda 分类: cs.CL
缺口
可穿戴传感产生的恰恰是”对 AI 助手应当有用”的那类数据：对生理与行为信号的连续监测，跨越数月，且是针对某一个个体的。然而现有基准很少检验一个 AI 系统能否对一位真实用户的「纵向可穿戴记录」做推理。
这种缺席是有后果的，而不只是”覆盖不足”，因为被跳过的那项能力，恰恰是让可穿戴设备变得有用的那一项。回答”我的静息心率是多少”是检索。回答”自从我的活动量下降之后，我的睡眠变了吗”则要求对用户自己的历史做推理——而那才是用户真正会问的问题。
可穿戴传感产生的恰恰是"对"的那类数据
对生理与行为信号的「连续监测」，跨越「数月」，
且针对「某一个个体」
|
v
[缺口] 现有基准「很少」检验一个 AI 系统能否对
一位真实用户的「纵向可穿戴记录」做推理
|
v
为什么这种缺席「有后果」
被跳过的那项能力，恰恰是让可穿戴设备变得「有用」的那一项：
"我的静息心率是多少？" -> 「检索」
"自从我活动量下降后，我睡眠变了吗？"
-> 要求对「用户自己的历史」做推理
<- 而那才是用户真正会问的问题
增量
一句话: 在这篇论文之前，没有基准检验过”对一位真实用户的纵向可穿戴记录做推理”；在这篇论文之后，由 200 位用户、每人最多 500 天构建的 4,084 道题，按推理类型划分，显示模型成绩在 19.6% 到 72.9% 之间、且多数低于 60%。
核心机制
构造上的选择比头条数字更要紧，因为基准的价值在于它是否测对了能力。
真实用户、真实记录：4,084 道十选项选择题，由 200 位真实用户的可穿戴时间序列、血液生物标志物与人口学信息构建，每位用户有最多 500 天的每日测量。也就是说，这些问题植根于个体历史，而不是总体统计或合成轨迹。
保留了真实的可穿戴分布，包括设备噪声与个体间差异。这个设计决定，才是把”真实的基准”与”干净的基准”区分开的东西。设备噪声与个体间变异恰恰是让纵向推理在部署中变难的东西；一个把它们抹平的基准，测量的是没人会去面对的部署条件。
16 种题型，沿两条互补的轴组织，而这两条轴就是诊断性的内容：
数据推理 vs 健康推理——把对纵向测量的计算与生理学解释分开。这是两项不同的能力，单一分数会把它们混在一起。一个模型可能把信号聚合对了、却解释错了，或者反过来；而只有这条轴能揭示是哪一种。
单信号推理 vs 跨信号推理——把关于单个信号的推理与多个信号的整合分开。跨信号推理才是一份纵向记录”不止是一组序列”的地方，而且它很可能难得多；这条轴把它定位出来。
一个”双重接地”框架被用来规模化地构造问题：由文献支撑的生理学发现，结合经统计验证、由人群数据支撑的生理学模式。两者都需要，只靠任一方都不够。文献接地确保问题反映真实的生理机制，而不是统计巧合；人群接地确保这些关系在这些用户的数据里确实存在，而不是从教科书里假设来的。正是这种配对，让问题能够批量生成却仍然有意义。
结果既验证了这个基准，也界定了它。在 14 个闭源与开源 LLM 上，成绩分布在 19.6% 到 72.9% 之间，而随机基线是 10%。由此得到三点：
- 这个基准能区分模型能力——在十选项格式上 3.7 倍的跨度，意味着这些问题既不是能猜出来的，也不是一致地超出所有模型能力的。
- 它远未被解决——在一个十选项选择题上，上限是 72.9%，也就是说最好的模型仍然错掉大约四分之一。
- 多数模型低于 60%——也就是说中位数系统离”有能力”比离”随机”更近；对任何考虑部署的人来说，这才是头条。
构造：真实用户、真实记录
4,084 道「十选项」选择题
由 200 位「真实用户」的可穿戴时间序列、血液生物标志物
与人口学信息构建
每位用户有最多 500 天的每日测量
-> 问题植根于「个体历史」，
而不是总体统计或合成轨迹
保留「真实分布」
包括「设备噪声」与「个体间差异」
<- 这个决定把"真实的基准"与"干净的基准"区分开
<- 它们恰恰是让纵向推理在部署中变难的东西；
抹平它们，测的就是没人会面对的部署条件
16 种题型、两条互补的轴
[轴一] 「数据 vs 健康」推理
对纵向测量的「计算」 vs 生理学「解释」
-> 单一分数会把两项不同能力混在一起
-> 模型可能聚合对了、解释错了，或反过来；
只有这条轴能揭示是哪一种
[轴二] 「单信号 vs 跨信号」推理
单个信号 vs 多个信号的「整合」
-> 跨信号才是"记录不止是一组序列"之处，
而且很可能难得多；这条轴把它「定位」出来
「双重接地」以支撑规模
由「文献」支撑的生理学发现
+ 经「统计验证、由人群数据支撑」的生理学模式
<- 两者都需要，只靠一方都不够：
文献 -> 问题反映真实的「生理机制」，而非统计巧合
人群 -> 这些关系在这些用户数据里「确实存在」，
而非从教科书假设而来
-> 让批量生成仍然「有意义」
结果（14 个闭源与开源 LLM）
19.6% 到 72.9%，随机基线 10%
- 3.7 倍的「跨度」说明问题既非可猜、也非一致超出能力
-> 它有「区分度」
- 上限 72.9%：仍错掉约四分之一
-> 「远未解决」
- 多数模型低于 60%：中位数系统离"有能力"
比离"随机"更近
可以用**“一场用你自己的通勤路线、而不是标准考场来出的驾照考试”来理解这件事： 标准考场告诉你一个人会不会开车。而你的通勤路线告诉他能不能开你这条路——那个让人犯迷糊的路口、下午三点的小学、会积水的那段。 论文坚持保留设备噪声与个体差异，是同一个直觉：有意思的失败不是”模型读不出心率”，而是”模型分不清真实趋势与噪声的一周**”。 而那两条轴做了单一分数做不到的诊断工作：把”算对了但解释错了”与”根本整合不了两个信号”分开——这是”可修的推理缺口”与”缺失的能力”之间的差别。
关键概念
- 对个体记录的纵向推理: 基准所隔离的那项能力，也是让可穿戴设备有用的那一项。它区别于”对某个信号的检索”，这也是为什么问题是关于历史而非数值的。
- 保留真实分布: 设备噪声与个体间差异被保留而非清洗。正是它让基准代表部署条件，而不是理想化的数据。
- 两条诊断轴: 数据 vs 健康推理、单信号 vs 跨信号。它们把单一准确率会混掉的能力分开——这才把一个分数变成一次诊断。
- 双重接地: 文献支撑的发现 + 经统计验证的人群模式。文献接地让问题保持生理性；人群接地让它们存在于数据中。两者合起来才允许规模不失意义。
- 以跨度作为效度证据: 十选项格式上 19.6%~72.9%。如果所有人都接近随机或接近满分，就没有区分度；这个范围有。
框架转变
之前（基准很少检验纵向记录）：
可穿戴传感产生数月生理数据
-> 评测聚焦于信号级任务
-> "对这位用户的历史做推理"未被检验
-> 部署场景中的能力缺口始终不可见
之后（一个接地、按轴划分的基准）：
4,084 道题，来自 200 位真实用户，每人最多 500 天
保留真实噪声与个体差异
16 种题型按推理种类划分
-> 模型成绩 19.6%~72.9%，随机基线 10%；多数低于 60%
-> 在选择题格式上远未解决
从”在信号级任务上评测模型、而部署要求历史级推理”，转变为”一个要求对真实个体记录、连同真实噪声做推理的基准”，核心转变在于：可穿戴设备所需的能力是纵向的、且以”某一个人”为条件——而它此前从未被测量。
专家评审
选题眼光: 极好，而且它瞄准的是一项部署需要、而评测跳过的能力。 可穿戴设备是语言模型自然的近期应用，恰恰因为数据是连续且个人化的；而这个领域一直在测量”那个个人连续性”之外的一切。
方法成熟度: 两条轴是最强的一环，因为它们把一个分数变成一次诊断——“聚合对了但解释错了”与”根本整合不了信号”是不同的问题，而单一数字说不出是哪一种。 保留设备噪声与个体间差异，是让基准对部署诚实的决定。 “双重接地”框架处理了”规模化生成问题而不漂向琐碎或巧合”这个真实困难；要求文献与人群两者同时接地，是正确的合取。
实验诚意: 十选项格式上 3.7 倍的跨度，是问题具备区分度的证据；而”多数模型低于 60%、最好的达到 72.9%“在两个方向上都是坦率的。 范围上的保留是：200 位用户对基准而言足够，但对”由人群数据支撑”的主张来说是偏小的人群，因此有些问题可能编码了这个队列特有的模式——论文自身的双重接地框架正是缓解措施，但”还剩多少队列特异性”没有被量化。
写作功力: 两条轴命名清楚，且各自给了一句理由——这才让基准可用而不只是可报告：你知道某个失败落在哪条轴上。 由于实际问题是”这个缺口有多少是可补的”，若能按轴各给一道例题，会让难度对不读完整题集的读者变得具体。
判决: 强接收（Strong Accept） — 它为”对个人可穿戴记录的纵向健康推理”构建了一个真实且有诊断力的基准；其轴能定位失败，其接地框架允许规模而不失生理学意义。
要点总结
- 测对一份记录做推理，而不是从记录里检索。“我的静息心率是多少”与”我活动量下降后睡眠变了吗”是不同的能力。
- 保留噪声。把设备伪影与个体间变异从健康基准里清洗掉，等于移除让部署变难的东西。
- 沿能力轴划分基准。数据 vs 健康、单信号 vs 跨信号，产生的是单一准确率会混掉的诊断。
- 让生成的问题双重接地。文献让它们保持生理性，人群统计让它们存在于数据中——只靠任何一方，都会得到一个什么都没测量的基准。