Paper: 2611.24778 — Counterfactual World Models: JEPAs That Reason About “What If” Authors: Randall Balestriero, Yann LeCun, Kerda Varaku Categories: cs.LG, cs.AI, cs.CV, cs.RO, stat.ML Reading scope: Original HTML, focusing on the main theoretical statements, experiments and limitations, with numerical checks against Tables 5 and 11. Figure captions were read; images, code and complete proofs were not independently verified.

The Problem

A boat misses its mooring. Was the heading wrong, or did the current carry it away? The paper uses this situation to expose a gap in ordinary world models: predicting the average destination under another heading does not answer where this boat would have arrived in the current it actually encountered.

Before the trip, averaging over possible currents is sensible. After the trip, the endpoint contains new information. Discarding it means throwing away evidence about this particular event. Balestriero and colleagues ask how a model can keep that evidence while changing only the action.

The Explanation

The missing information is not just another training example

Section 2.2 shows why even perfect factual predictions are insufficient in general. A dataset can reveal the possible destinations under each heading without revealing which destinations share the same hidden current. Different physical explanations can agree on every observed distribution and disagree about that pairing. More capacity alone cannot identify it.

The authors therefore add structure, not a claim that the model has somehow learned every possible past. In suitable physical systems, propulsion and an external current enter through separate channels. Over a short interval, changing the heading then changes the endpoint by nearly the same amount across currents. Section 3.3 derives the approximation and its error bounds; smoothness by itself is not enough.

Keep the observed miss, replace the predicted action effect

Definition 3.1 gives a small construction:

  1. Ask the existing world model where the executed action was expected to end.
  2. Compare that prediction with the actual endpoint: this residual records what happened differently in this event.
  3. Add the same residual to the model’s prediction for the alternative action.

Equivalently, start at the actual endpoint and move by the model’s predicted difference between the two actions. Figure 1 illustrates this relationship. The counterfactual correction adds no parameters or separate counterfactual training stage, though a useful factual predictor must already exist.

The experiments use JEPAs, or joint embedding predictive architectures. An image encoder turns a frame into compact numerical features, and a predictor estimates future features instead of reconstructing every pixel. The correction can act on these features or on physical quantities read from them.

For the boat, this carries the observed drift into the alternative heading. It works exactly when the disturbance contributes the same shift under both actions, and approximately when their interaction is small. It can fail when a different heading takes the boat into substantially different water.

Test the event information, not just the model

Section 4.2 compares predictors sharing the same learned weights. In the Table 5 experiment, an image encoder called MAE-B turns frames into compact features. Its weights stay fixed while the transition predictor is trained, so the compared corrections receive the same visual representation. With one input frame and an 8k-episode factual collection, the HandReach simulator’s ordinary predictor explains 38.1% of counterfactual outcome variance. Correcting after a physical readout raises that to 85.6%, with a reported bootstrap interval of 83–87%. The score measures how much outcome variation the predictions account for; it is not an 85.6% task-success rate.

The readout matters: it translates image features into physical quantities and is fitted using state labels from factual frames. Correcting directly in feature space needs no such labels for the correction itself and reaches 82.4% in that comparison. The largest reported gains should not be described as entirely free of state supervision.

Learning the action difference still takes data. For BlueROV2 with MAE-B, Table 5 reports 73.6% after readout correction with an 8k-episode factual collection and 93.5% with a collection sixteen times larger. The arithmetic is unchanged; its predictions improve.

Why a learned model can beat a simulator reference

Section 4.4 reports 35–69% success in hindsight action selection, versus 14–23% for a planner with the true simulator but no observed outcome. This is an information advantage: the counterfactual model sees the event that revealed the disturbance. The planner knowing the realized disturbance succeeds in every tested event.

The goals are attainable by construction, and the chosen action is scored by replaying the same source state and disturbance in simulation. These results do not establish that the correction beats a perfect simulator or solves general prospective robot planning.

Core Concepts

An individual counterfactual holds the event fixed while changing the action. Asking about the boat under the same current differs from asking what usually happens under that heading.

Action–disturbance interaction measures whether the action change behaves differently under different disturbances. An offset that pushes every command sideways is favorable. A worn motor that scales each command is different: doubling the command changes the effect of its hidden weakness, so one fixed residual can mislead.

The coordinates of correction determine what “add the residual” means. Adding image pixels can leave ghosts rather than move a boat. Section 4.3 shows that compact features or physical readouts can behave very differently from raw pixels; accurate counterfactuals in selected coordinates do not imply a faithful counterfactual image of the whole world.

The Insight

An outcome can supply information that a better average predictor still lacks. Here, the useful division of labor is precise: the observation carries evidence about this event, and the model supplies the effect of changing the action. Its value depends on whether those two contributions can be separated.

A Critical Reading

The paper connects an impossibility result, a constructive rule and measured failures. Its conditions deserve as much attention as the gains. Separate disturbance channels, appropriate coordinates and a suitable transition duration are assumptions to check; contacts, saturation and multiplicative faults can violate them. Table 11 shows severe deterioration under a pure gain fault rather than hiding that case.

There is also a causal identification requirement. Section 3.4 assumes the logged action does not exploit relevant information unavailable to the model, and that the queried alternative actions have support in the data. A skipper who feels an unrecorded current can break the first condition. A controller that never makes a hard turn cannot justify an estimate for one merely by collecting more straight trips.

All reported environments are simulated. Disturbances are controlled, quantities are prespecified, and physical readouts use factual state labels. The experiments support this mechanism under those protocols; they do not yet establish reliability on real robots or long counterfactual rollouts.

What You Can Use

If you already have a world model, retain the executed action, its predicted outcome and its observed outcome. That record lets you test whether a residual helps evaluate alternatives for the same event. Validate the separation assumption with a simulator or paired trials before relying on it.

Return to the missed mooring: the useful question is whether another heading would have helped in that current. Carrying yesterday’s miss into tomorrow’s unrelated gust is a different problem. Section 4.4 finds that later decisions can benefit when the disturbance persists; a fresh, independently drawn disturbance does not guarantee the same benefit.

论文: 2611.24778 — Counterfactual World Models: JEPAs That Reason About “What If” 作者: Randall Balestriero, Yann LeCun, Kerda Varaku 分类: cs.LG, cs.AI, cs.CV, cs.RO, stat.ML 阅读范围: 阅读原始 HTML 中的主要理论陈述、实验与局限,并对照表 5、表 11 核查关键数字。阅读了图注,未独立核验图像、代码及完整证明。

问题

船没靠上码头,究竟是方向选错了,还是水流把它推偏了?论文用这个场景指出:世界模型即使知道换个方向通常会漂到哪里,也未必知道刚才那条船遇上同一股水流时,换个方向会怎样。

出发前,不知道会遇上哪股水流,预测平均结果很合理。 可船已经走过一趟,实际终点就多了一条线索:这次水流究竟造成了什么影响。 再问一遍普通模型,却把这条线索扔掉,等于又回到了出发之前。 Balestriero 等人要做的,是留住刚才这件事,只替换其中的动作。

翻译

缺的不是再多看一条记录

第 2.2 节先说明,单靠通常的观测记录,这个问题一般没有唯一答案。 我们可以知道每种航向下终点各有什么概率,却不知道两种航向的哪些终点对应同一股水流。 不同的物理解释可能完全符合已有记录,却对“刚才换个方向”给出不同答案。 模型更大,并不会自动补上这层对应关系。

作者因而引入额外的物理条件。 如果船的动力和外部水流通过各自的通道起作用,那么在足够短的时间里,换方向带来的位移变化,就可能基本不受这次水流的影响。 第 3.3 节推导了这个近似及其误差边界。 这里需要的是分开的作用通道,不能只说系统变化平滑就够了。

留住这次偏差,替换动作的影响

定义 3.1 给出的办法很短。 先问模型:按照刚才执行的动作,本来预计会到哪里?再看实际到了哪里,两者之差就是这次预测留下的偏差。 最后,让模型预测另一个动作,把刚才那份偏差加上去。

也可以从实际终点出发,只移动模型认为“两个动作之间差了多少”的那一段距离。 图 1 展示了这层关系。 这步修正不增加参数,也不用另设一轮反事实训练;但它当然仍需要一个已经能做有效预测的模型。

实验用的模型先把画面压成一组紧凑的数字特征,再预测下一刻的特征,而不逐像素重画未来。 这类方法叫共同嵌入预测架构,简称 JEPA。 修正可以发生在这些特征上,也可以发生在从特征读出的位置、速度等物理量上。

回到船上,就是把这次水流造成的漂移,带到另一个航向的预测中。 如果水流对两个动作都增加同一份位移,这个办法可以精确成立。 两者相互影响较小时,它近似成立。 可如果换个航向就驶入很不一样的水流,原样搬过去的偏差就可能错了。

改善来自这次事件的信息

第 4.2 节把同一套模型权重分别用于普通预测和反事实修正。 表 5 的一项实验用名为 MAE-B 的图像编码器,把画面转成紧凑特征。 训练下一刻的预测器时,这个图像编码器的权重保持不变,因此参与比较的修正方式收到相同的视觉表示。 HandReach 模拟器的这组比较使用一帧输入及 8k 段事实轨迹。 普通预测器能解释反事实结果方差的 38.1%;经过物理量读出后再修正,达到 85.6%,论文给出的自助采样区间为 83%–87%。 这是“解释了多少结果差异”的指标,不能读成任务成功率达到 85.6%。

读出环节也不是没有代价。 它把图像特征转成位置、角度等物理量,需要事实画面上的状态标签来训练。 同一项比较中,直接在特征上修正达到 82.4%,修正本身不需要这些状态标签。 因此,不能把最高的数字概括成完全不需要状态监督。

动作差异本身也得学准。 表 5 的 BlueROV2 模拟中,MAE-B 模型用 8k 段事实轨迹时,读出后修正达到 73.6%;数据增至原来的十六倍时达到 93.5%。 修正的算术没有变化,变好的是它依赖的预测。

为什么会胜过一个知道模拟器的参照

第 4.4 节还有一个容易读错的结果:事后选动作时,反事实模型的成功率为 35%–69%,知道真实模拟器、却没看见刚才结果的规划器为 14%–23%。 区别在于谁看到了这次事件透露的信息。 知道真实扰动的规划器,在测试事件中全部成功。

这些目标本来就设计成可以到达,所选动作也在同一初始状态、同一扰动下重放打分。 这说明观察结果能帮助复盘同一件事,不能据此宣布学出来的模型胜过精确模拟器,更不能直接推广成一般的机器人事前规划能力。

核心概念

个体反事实,是留住同一件事,只换其中的动作。 船遇上同一股水流时换个航向,与另一趟航行通常会怎样,问的不是一回事。

动作与扰动的相互作用,问的是:换动作带来的变化,会不会随扰动而变?不管什么指令都多推偏一点,比较适合搬用同一份偏差。 电机老化、只能交付指令的一部分,就不同了:指令翻倍,故障造成的差距也会跟着变,固定修正反而可能误导。

修正所在的坐标,决定了“把偏差加回去”究竟在加什么。 直接加图像像素,可能留下重影,却没有把船移到正确的位置。 第 4.3 节说明,紧凑特征、物理量读出和原始像素的表现可以很不一样。 几个选定物理量预测准确,不等于已经生成了一幅忠实的反事实世界图像。

洞见

实际结果里,有些信息是更好的平均预测也替代不了的。 这篇论文的分工很清楚:观察告诉我们刚才那件事发生了什么,模型提供换动作会改变多少。 能不能把两者分开,决定了这条捷径有没有用。

博导审稿

论文把不能识别的问题、可以采用的构造以及明确的失败条件接在一起,这是值得认真看的地方。 但物理条件应当和改善数字一起读:扰动要有独立的作用通道,坐标要合适,所选时间段也要让相互影响足够小。 接触、饱和或随指令缩放的故障都可能破坏它。 表 11 里,纯增益故障会让修正后的表现大幅下降,并不是所有“预测错了”都适合把偏差搬过去。

另一个前提是因果识别。 第 3.4 节要求,记录中的动作不能利用模型看不到的相关信息,而且数据得覆盖要询问的替代动作。 如果船长感到了水流、据此打舵,却没把这条信息记下来,第一个条件就可能不成立。 如果控制器从来不急转弯,多积累直航记录也无法自动证明急转弯会怎样。

所有实验环境都是模拟器,扰动经过控制,打分物理量事先选定,物理读出还使用了事实状态标签。 这些实验支持对应协议下的机制,尚不能证明它在真实机器人或长时间反事实推演中可靠。

启发

如果已经有一个世界模型,可以把执行的动作、原先的预测和实际结果一起留下。 这组记录提供了一个具体试验入口:刚才的偏差,能不能帮助评估同一事件里的其他动作?正式依赖它之前,先在模拟器或可以固定扰动的配对试验中核查条件。

船刚才没靠上码头,可以追问同一股水流下怎样打舵会更好。 把昨天那次漂移搬到明天另一阵风里,则是另一个问题。 第 4.4 节也发现,扰动持续存在时,后续动作可能受益;但换成新抽取、彼此独立的扰动,同样的好处就没有保证。