Paper: 2611.26627 — RL Post-Training in Diffusion Transformers Is Activation Steering
Authors: Tuna Han Salih Meral, Kaan Oktay, Alec Helbling, Atakan Kara, Duen Horng Chau, Pinar Yanardag
Category: cs.CV, cs.LG
Reading scope: Original arXiv v1 HTML, including methods, results tables, and limitations. Figure captions were read; figure images were not visually inspected. The experiments were not reproduced.
The Problem
Ask an image generator to put a short phrase on a poster, and it may turn the letters into decorative noise. Ask for a particular arrangement of objects, and some may end up in the wrong place. Reinforcement learning can improve these results by rewarding outputs that score better. The usual explanation is that the model learned a more elaborate, prompt-specific transformation: the right change depends on what each prompt asks for.
Tuna Han Salih Meral and colleagues test a smaller possibility. What if much of the improvement can be reproduced by adding the same small shift inside the model, regardless of the prompt? If so, that would show such a shift is sufficient to recover much of these checkpoints’ measured gains. It would not show that reinforcement learning itself made only this change, or that every task can be solved with one.
The Explanation
Add the same nudge at each step
During image generation, a diffusion transformer repeatedly updates a noisy image toward a finished one. In each block, information passes through internal channels, represented as vectors of numbers. The authors add a learned steering vector to a block’s output. Every token in that stream receives the same vector: the word token, the image patches, and every prompt using that model.
There are a few small switches. One vector can be used for the conditional or unconditional pass in classifier-free guidance; another can be used in an early or late part of the sampling schedule; image and text streams can have separate vectors. These choices let the same kind of nudge differ by where it enters the model, without depending on the prompt or the token (§3.2).
For SD3.5-M, the full set contains 288,768 parameters, against 18.8 million in the rank-32 LoRA used by the corresponding PickScore-tuned checkpoint—about 65 times fewer. The model’s weights remain frozen. The question is whether these small additions can reproduce what the public, already RL-tuned model does.
Match the model’s movement, not its reward
To fit a nudge, the team first collects generated samples from the RL-tuned model. They then give the same noisy image state, prompt and sampling step to both models. The steering vectors are adjusted so the frozen base model predicts the same next update—called the velocity—as the RL-tuned model. The objective compares these updates; it does not use the reward score. Both models stay frozen during this fitting (§3.3).
This distinction matters. The fitted vectors imitate the behavior of an RL checkpoint through a simpler interface; they are not being taught directly to maximize OCR, image preference, or another reward. The reward scores are held back to test how much of the checkpoint’s measured gain the imitation recovers.
On six public image checkpoints, the fitted vectors recover 90.0–96.3% of each checkpoint’s gain on its evaluated reward and test set (Table 1; §4.2). For example, SD3.5-M’s PickScore-tuned model gains 1.560 raw points over the base model, while the fitted vectors gain 1.416, or 90.8% of that measured improvement. For the OCR-tuned checkpoint, the vectors recover 94.4%; for GenEval, which scores object-composition prompts, they recover 95.9%.
An unfitted average shift is a weaker baseline. On the same SD3.5-M OCR test, its mean-difference vectors recover 21.5% of the checkpoint gain, versus 94.4% after fitting. A useful direction in the activation changes is not enough by itself; the learned scale and where the shift enters the model matter (§4.3; Table 1).
The paper also tests two video checkpoints. On Wan2.1, the vectors recover 111.9% of the RL checkpoint’s gain on HPSv3; on Wan2.2, they recover 89.8% on a rule-based score. A value above 100% means the fitted version scored higher than that checkpoint on this test measure, not that it reproduces the checkpoint exactly. Crucially, only two of six video RL checkpoints evaluated had a positive gain on the reward used for comparison. The authors limit the video claim to those two models (§4.1; Table 1 and Appendix I).
Training the small shift is a separate test
The fitted vectors above copy a checkpoint that already exists. The authors also ask a different question: can reinforcement learning learn useful vectors from scratch? They initialize the vectors at zero, freeze SD3.5-M, and run Flow-GRPO training on the vectors alone. This training does see the reward, unlike the velocity-matching fit.
On the PickScore and OCR tests, this separately RL-trained version reaches 103.6% and 104.6% of the corresponding full-checkpoint reward gains (Table 3; §4.6). In a pairwise study across 50 prompts and 25 raters, the vector-steered images beat the base model 60–65% of the time, counting ties as half a win; against the full RL-tuned model, they were about even at 50–52%. The study supports the measured comparison on this sample, not a universal judgment about image quality.
Core Concepts
A steering vector is a repeated nudge. It adds the same list of numbers to every token at a chosen block output. The block still processes different prompts differently, so the downstream effect need not look the same for each image.
Velocity matching is imitation. It teaches the small shift to make the frozen model’s next denoising update resemble a particular RL checkpoint’s update. No reward is optimized in this fit.
Reward training is a separate route. Starting the vectors at zero and updating them with Flow-GRPO lets reward feedback train them directly. That experiment tests whether the small parameterization can learn an improvement, rather than whether it can imitate an existing checkpoint.
The Insight
An elaborate model update can have much of its visible reward effect reproduced by a much smaller internal adjustment. The experiments establish sufficiency for the tested checkpoints and scores; they do not identify every change that reinforcement learning made.
A Critical Reading
The image results span six public checkpoints, several reward types, and held-out test prompts. They provide evidence beyond a single image model, but the recovery percentages are defined relative to each checkpoint’s own measured reward gain. They compare scores on specific test sets; they do not mean that each generated image matches its RL counterpart.
The authors explicitly stop short of claiming necessity: an RL-tuned model need not itself use these vectors to obtain its gain. Nor do the vectors show that RL updates are generally low-rank. The video evidence is narrower still, covering two positive-gain checkpoints on two model families, not video reinforcement learning at large. Finally, the supervised fine-tuning comparison is a useful boundary: for teaching a model a fictional person’s identity it could not already draw, a same-size LoRA recovered more of the gain than the steering vectors (Table S7; §4.5).
What You Can Use
If a diffusion model’s reward score improves after fine-tuning, ask whether the gain depends on prompt-specific weight changes. A useful diagnostic is to fit a small, fixed intervention against the tuned model’s denoising updates, then evaluate both on held-out prompts and the original reward. If the intervention recovers the score, it is evidence that the tested gain can be expressed compactly. It is not evidence that the model learned nothing else, or that the same shortcut will work on a new task.
论文:2611.26627 — RL Post-Training in Diffusion Transformers Is Activation Steering
作者:Tuna Han Salih Meral、Kaan Oktay、Alec Helbling、Atakan Kara、Duen Horng Chau、Pinar Yanardag
分类:cs.CV、cs.LG
阅读范围:阅读 arXiv v1 原始 HTML 中的方法、结果表和局限;也阅读了图注,但没有查看图像本身;未复现实验。
问题
让图像模型在海报上写一句短话,它可能把字画成花纹;让它摆好几件东西,物体又可能各就各位不准。强化学习可以给生成结果打分,再用奖励把结果往高分方向推。人们很容易以为:模型要变好,就得学会一套复杂的、因提示词而异的改变;每种需求,都得有一套对应办法。
Tuna Han Salih Meral 等人检验了一个更简单的可能:改善是否有相当一部分,可以靠在模型内部加一个固定偏移来复现?如果可以,说明这个偏移足以重新做出这些检查点在指定指标上的大部分增益。可这并不能证明强化学习本身只做了这一种改动,也不能说所有任务都能靠它解决。
翻译
每一步都推同一个方向
扩散模型生成图像时,会反复把一张带噪图像往成品方向更新。每一层都会输出一些内部信息,可以看成一串数字向量。作者在某层的输出上加一条学出来的“引导向量”。这一串数字会同时加到这个信息流里的每个 token:文字、图像块,以及模型接到的每种提示词。
这项偏移也有几处开关:分类器自由引导里的条件与无条件两趟可以各用不同向量;采样早期和后期可以各用一组;文字流和图像流也能分开设。向量因此能随它进入模型的位置和采样阶段而变化,但不会因提示词或具体 token 而变化,见第 3.2 节。
以 SD3.5-M 为例,这套向量共有 288,768 个参数;对应的 PickScore 微调模型使用 1,880 万参数的秩 32 LoRA,前者少约 65 倍。模型原来的权重保持冻结。作者想知道:只加这么一小组偏移,能不能复现已经经过强化学习的公开模型做出的事?
拟合的是模型的动作,不是奖励
作者先收集强化学习检查点生成的样本。随后,对同一张带噪图像、同一段提示词和同一步采样,两边模型各算一次下一步更新,也就是速度预测。训练的只是一组引导向量,目标是让冻结的基础模型也做出和强化学习模型相近的更新。这个拟合目标比较的是两次更新,没有把奖励分数放进目标函数;两边的模型权重都不变,见第 3.3 节。
这个区别很重要。拟合出来的向量,是用更简单的接口去模仿已经存在的强化学习检查点;它不是直接被教着提高 OCR、图像偏好或其他奖励。奖励分数留到测试时,用来衡量模仿复现了多少已有增益。
六个公开图像检查点的结果显示,引导向量复现了各模型在所测奖励和测试集上 90.0%—96.3% 的增益(表 1、第 4.2 节)。例如,SD3.5-M 的 PickScore 检查点比基础模型多 1.560 个原始分数;向量版本多 1.416,相当于这项增益的 90.8%。OCR 检查点复现了 94.4%,GenEval 检查点复现了 95.9%;GenEval 用来评测模型按提示词安排物体的能力。
不经训练、直接取平均变化,效果就弱得多。同一组 SD3.5-M OCR 测试里,这种“平均差分向量”只复现了检查点增益的 21.5%;拟合后的向量达到 94.4%。从内部变化取一个有用方向还不够,还得学好它的大小,以及把偏移放在哪里,见第 4.3 节和表 1。
论文也测了两个视频模型检查点。Wan2.1 的向量在 HPSv3 上复现了 111.9% 的强化学习增益;Wan2.2 在一项规则评分上复现了 89.8%。超过 100% 的意思是:在这项测试指标上,拟合后的版本分数高于检查点,并不代表它逐项复制了检查点。更要紧的是,作者评测的六个视频强化学习检查点中,只有两个在被比较的奖励上有正增益。论文把视频结论严格限定在这两个模型上,见第 4.1 节、表 1 和附录 I。
另一条路:让强化学习只训练这组向量
上面的拟合是在模仿已经完成的强化学习检查点。作者又问了另一个问题:如果从零开始,强化学习能否自己学出有用的偏移?他们把向量初始化为零,冻结 SD3.5-M,只用 Flow-GRPO 训练这组向量。这次训练会收到奖励反馈,与前面的速度拟合不同。
在 PickScore 和 OCR 测试上,这个单独经过强化学习的向量版本分别达到了相应完整检查点增益的 103.6% 和 104.6%(表 3、第 4.6 节)。另一项成对比较覆盖 50 个提示词和 25 位评审;把平局折半计入胜场后,向量图像战胜基础模型的比例为 60%—65%,与完整强化学习模型相比则为 50%—52%,基本持平。这项小规模用户研究支持的是这组样本里的比较,不是对所有图像质量的普遍判断。
核心概念
引导向量是重复加上的一推。 它在指定层输出上,把同一串数字加到每个 token。下游的模型仍会因不同提示词而作不同响应,所以同一偏移不等于每幅图都会发生同样的变化。
速度匹配是在模仿。 它训练一小组偏移,让冻结模型的下一步去噪更新接近某个强化学习检查点的更新。这个拟合过程不优化奖励。
奖励训练是另一条路。 从零初始化向量,再用 Flow-GRPO 更新,才是直接拿奖励反馈训练这些向量。它回答的是这种小参数形式能不能学会提升,而不是它能不能模仿已有检查点。
洞见
强化学习检查点的许多可测增益,可以由小得多的内部调整复现。实验说明这种形式在所测检查点上够用,却没有找出强化学习到底做了所有哪些改变。
博导审稿
图像结果覆盖六个公开检查点、几类奖励和留出的测试提示词,不只盯着单一模型。但“复现百分比”的分母是每个检查点自身测得的奖励增益。它表示两种方法在特定测试集上的分数接近,不表示每一张生成图都一样。
作者也明确没有声称这种方法是必需的:强化学习模型自身未必通过这些引导向量获得增益;少量向量也不意味着强化学习更新普遍是低秩的。视频证据更窄,只涵盖两个模型家族里各一个、且在目标奖励上取得正增益的检查点,不足以概括视频强化学习。监督微调则划出另一条边界:让模型学会画出原本不会画的虚构人物时,同参数量的 LoRA 比引导向量恢复得更多,见表 S7 和第 4.5 节。
启发
如果图像模型微调后奖励分数上升,可以追问:这份增益是否必须依赖提示词特定的权重改变?一个诊断办法,是让小型固定干预去拟合微调模型的去噪更新,再在留出的提示词和原奖励上比较。如果分数追回来了,说明所测的增益可以被紧凑表达;这不能证明模型别的什么都没学,也不能保证下一种任务仍有同样捷径。