GenRM-Flow: Video Generators Inherently Function as Process-Aware Reward Models in Flow Matching

Paper: 2611.11421
Authors: Siming Fu, Ruizhe He, Zheming Fu, Zeyue Xue, Fengkai Liu, Yufei Jiao, Jie Huang, Xiaoxiao Ma, Xuan Yang, Lihui Luo (Zhejiang University, Joy Future Academy, Xiamen University, Tsinghua University)
Categories: cs.CV

The Gap

Aligning diffusion and flow-matching video generation models with human aesthetic, motion, and temporal preferences is the primary bottleneck in generative media. In text models, reinforcement learning from human feedback (RLHF) utilizes scalar reward models (RMs). However, directly transplanting reward modeling to continuous video synthesis creates severe structural problems:

1. The VLM Representation Paradigm (Pixel Space): Existing approaches (e.g., VideoAlign) decode generated noisy latents into pixel frames via a Variational Autoencoder (VAE) and feed them to an external Vision-Language Model (e.g., Qwen3-VL).
   The Breakdown: VAE decoding fundamentally fails in high-noise regimes ($t \in [0.8, 1.0]$). As a consequence, VLM rewards can only be computed at the very final generation steps, leaving the critical early-stage global composition and gross motion dynamics completely unsupervised.
2. The Auxiliary Scorer Paradigm (Feature Space): Other frameworks (e.g., PAVRM) freeze generator backbones and train separate classification or regression heads on intermediate features.
   The Breakdown: This places scoring in a disjoint parameter space. Parametric divergence between the scoring head and the generator leads to severe out-of-distribution hallucinations and reward hacking during online reinforcement learning.
3. The Core Paradox: The generator’s most potent capability—predicting continuous vector velocity fields and denoising high-dimensional latents—is entirely discarded during evaluation.

                    THE TRADITIONAL VIDEO REWARD PARADOX

  [Paradigm A: External VLM Scorer]
  Noisy Latent x_t ──> VAE Decoder ──> Collapses at high noise! (Only works at t ≈ 0)
                                   ──> External VLM Evaluator (Expensive, slow, disjoint)

  [Paradigm B: Auxiliary Feature Head]
  Noisy Latent x_t ──> Frozen DiT Backbone ──> Separate Regression Head (Parametric Drift!)

  [The Root Question]:
  Why train separate networks when the generator already knows how to denoise velocity fields?

---

The Increment

One sentence: The authors present GenRM-Flow, proving mathematically that under KL-constrained preference optimization, an aligned flow-matching generator’s own relative difference in velocity-prediction error acts as an exact, native process reward model in latent space, eliminating auxiliary networks, VAE decoding, and external VLM scorers.

                   GENRM-FLOW: GENERATION AS PROCESS REWARD

  Input Prompt c + Video Latent Trajectory x_t
       │
       ▼
  Run identical forward pass on:
  1. Base Reference Policy:  v_{θ_ref}(x_t, t, c)  ──> Loss: L_{ref} = || v_{θ_ref} - u_t ||^2
  2. Preference-Tuned Model: v_{θ^★}(x_t, t, c)    ──> Loss: L_{θ^★} = || v_{θ^★} - u_t ||^2
       │
       ▼
  [Exact Latent Process Reward]:
  r_t(x_t, c) = β * [ L_{ref}(x_t, c) - L_{θ^★}(x_t, c) ]
       │
       ├──> Lower error in tuned model? ──> Positive reward for human preference!
       ├──> Evaluates directly in continuous latent space across ALL timesteps t ∈ [0, 1]!
       └──> NO auxiliary reward head, NO pixel VAE decoding, ZERO parametric drift.

---

Core Framework & Mechanisms

1. Mathematical Equivalence: From KL Optimal Policy to Velocity Error

Consider continuous-time flow matching, interpolating clean latent $x_0$ and Gaussian noise $x_1 \sim \mathcal{N}(0, I)$: $x_t = (1 - t)x_0 + t x_1, \quad u_t(x_t | x_0) = x_1 - x_0, \quad t \in [0, 1]$ The flow-matching velocity loss is $L_\theta(x_t, c) = \| v_\theta(x_t, t, c) - u_t(x_t | x_0) \|^2$. A known variational property of probability flows connects velocity loss to data log-likelihood: $-\log p_\theta(x_0 | c) \lesssim \mathbb{E}_{t, x_1} [ L_\theta(x_t, c) ] + \text{const}$

Under KL-constrained offline preference optimization, any objective (DPO, IPO, RWR) targets the analytical optimal policy: $\pi^*(x|c) = \frac{1}{Z(c)} \pi_{\text{ref}}(x|c) \exp\left( \frac{R_D(x, c)}{\beta} \right)$ Algebraically isolating the target preference reward $R_D(x, c)$: $R_D(x, c) = \beta \log \frac{\pi^*(x|c)}{\pi_{\text{ref}}(x|c)} + \beta \log Z(c)$

Substituting the negative velocity error $-L_\theta$ as the tractable surrogate for log-likelihood, the empirical reward emerges deterministically: $\hat{r}(x, c; \theta^\star, \theta_{\text{ref}}) \approx \beta \mathbb{E}_{t, x_1} \left[ L_{\text{ref}}(x_t, c) - L_{\theta^\star}(x_t, c) \right] + \beta \log Z(c)$

The preference score is simply the denoising advantage: if the aligned generator $\theta^\star$ predicts the vector field with smaller Euclidean error than the reference generator $\theta_{\text{ref}}$, the latent is structurally preferred!

                  UNIFIED REWARD EXTRACTION FROM OBJECTIVES

  Objective F             Family                Loss Formulation L_F(θ)
  ────────────────────────────────────────────────────────────────────────────────
  RWR (Reward Weighted)   Likelihood            E_{t,x_1} [ L_θ(x_w, t) ]
  IPO (Identity-Margin)   Non-saturating        E_{t,x_1} [ ( (L_ref - L_θ)_{x_w} - (L_ref - L_θ)_{x_l} - 1/(2τ) )^2 ]
  DPO (Bradley-Terry)     Contrastive           -E_{t,x_1} [ log σ( β [ (L_ref - L_θ)_{x_w} - (L_ref - L_θ)_{x_l} ] ) ]
  ────────────────────────────────────────────────────────────────────────────────
  All three objectives fundamentally calibrate the scalar discrepancy [L_ref - L_θ]
  into a calibrated preference ranking function!

2. Closed-Loop Process RL via FlowAWR

To drive online reinforcement learning, GenRM-Flow is integrated into Flow Advantage-Weighted Regression (FlowAWR):

1. Rollout: Policy $\theta_{\text{old}}$ rolls out $G$ sample trajectories conditioned on prompt $c$ using an ODE solver, yielding final latent $\hat{x}_0^{(i)}$.
2. Intermediate Process Reward: At sampled timestep $t$, forward-noise $\hat{x}_0^{(i)}$ into $x_t^{(i)}$ and compute the instantaneous velocity discrepancy: $r_t(x_t^{(i)}, c) = \beta \left[ L_{\text{ref}}(x_t^{(i)}, \hat{x}_0^{(i)}, c) - L_{\theta^\star}(x_t^{(i)}, \hat{x}_0^{(i)}, c) \right]$
3. Group Softmax Normalization: Neutralizes the intractable prompt-dependent partition function $\beta \log Z(c)$: $\tilde{r}_t(x_t^{(i)}, c) = \frac{\exp(r_t(x_t^{(i)}, c))}{\sum_{j=1}^G \exp(r_t(x_t^{(j)}, c))}$
4. Rectified Velocity Update: With advantage $A_t = G \tilde{r}_t - 1$, regress current velocity $v_\theta$ toward the advantage-weighted trajectory target.

---

Benchmark Results & Empirical Findings

1. Pairwise Preference Accuracy Across Noise Timesteps

The authors benchmarked GenRM-Flow against the leading video reward models: VideoAlign (VLM scorer) and PAVRM (auxiliary feature head) on Wan2.1-T2V across 5 noise intervals:

Model Backbone	Reward Model	Timestep $t \in [0, 0.2]$	$t \in (0.2, 0.4]$	$t \in (0.4, 0.6]$	$t \in (0.6, 0.8]$	High Noise $t \in (0.8, 1.0]$	Average Acc (%)
Qwen3-VL-8B	VideoAlign [28]	—	—	—	—	—	78.26
Wan2.1-1.3B	PAVRM [36]	77.19	77.50	78.27	76.11	71.25	76.06
Wan2.1-1.3B	GenRM-Flow (Ours)	87.77	88.08	88.49	88.61	81.00	86.79 (+10.73)
Wan2.1-14B	PAVRM [36]	81.87	83.85	84.79	84.69	78.02	82.64
Wan2.1-14B	GenRM-Flow (Ours)	89.10	89.94	90.46	90.75	84.55	88.96 (+6.32)

Key Result: In the critical high-noise interval ($t \in [0.8, 1.0]$) where structural layouts and motion trajectories coalesce, GenRM-Flow maintains 84.55% accuracy, whereas PAVRM collapses to 78.02% and VLM-based pixel scorers cannot evaluate at all.

2. Downstream Video Generation Quality (VBench Benchmark)

When used as the reward signal for FlowAWR reinforcement learning, GenRM-Flow achieves state-of-the-art results across both in-domain and out-of-domain VBench suites:

Base Model	Fine-Tuning Reward	Motion Smoothness (MS) $\uparrow$	Dynamic Degree (DD) $\uparrow$	Subject Consistency (SC) $\uparrow$	Human Action (HA) $\uparrow$
Wan2.1-1.3B	Pretrained Baseline	98.52	65.19	97.56	94.00
Wan2.1-1.3B	VideoAlign (VLM)	98.73	67.52	97.64	95.32
Wan2.1-1.3B	PAVRM	98.71	67.83	97.52	94.68
Wan2.1-1.3B	GenRM-Flow (Ours)	98.95	73.46	97.70	96.25
Wan2.1-14B	Pretrained Baseline	98.30	65.46	97.52	95.40
Wan2.1-14B	VideoAlign (VLM)	98.51	70.36	97.60	95.62
Wan2.1-14B	PAVRM	98.41	71.82	97.68	96.18
Wan2.1-14B	GenRM-Flow (Ours)	99.03	74.87	97.73	97.73

On the flagship Wan2.1-14B model:

• Dynamic Degree (DD) jumps from 65.46 to 74.87, introducing rich, complex fluid motion without blur.
• Human Action (HA) reaches 97.73, avoiding the limb distortion and spatial tearing that plagued PAVRM.

---

Practical Implications

1. Elimination of the Multi-Model Video Alignment Stack: Teams fine-tuning open video models (e.g., Wan2.1, LTX-Video, CogVideoX) no longer need to stand up separate VLM inference endpoints or train auxiliary discriminator heads. The generator’s own LoRA checkpoint serves simultaneously as the production generator and the RL process judge.
2. True Full-Horizon Process Rewards: Because flow-matching velocity loss is well-defined for all continuous timesteps $t \in [0, 1]$, reward credit assignment can guide early trajectory layouts, eliminating the “late-stage-only” bias of pixel reward models.
3. Zero Parametric Divergence: By evaluating rewards directly in the identical latent representation space of the base DiT, GenRM-Flow completely circumvents the out-of-distribution adversarial exploits that routinely derail video RLHF.

                    传统视频偏好对齐的割裂式管线

  【范式 A：外挂 VLM 像素级评分】
  加噪隐变量 x_t ──> VAE 图像解码 ──> 高噪区无法解码崩溃！(仅 t ≈ 0 有效)
                                  ──> 外部 VLM 打分 (推理成本高昂、动作理解迟钝)

  【范式 B：外置辅助特征打分头】
  加噪隐变量 x_t ──> 冻结 DiT 主干 ──> 外挂回归预测头 (参数空间漂移，极易奖励作弊)

  【根源命题】：
  流匹配生成器本身就精通全过程去噪速度预测，为何弃用其本体能力而舍近求远？

                   GENRM-FLOW：速度预测误差直出过程奖励

  输入提示词 c + 任意时刻加噪视频隐变量 x_t
       │
       ▼
  输入流匹配 DiT 网络执行两次并行前向计算：
  1. 冻结参考基座模型: v_{θ_ref}(x_t, t, c)  ──> 误差: L_{ref} = || v_{θ_ref} - u_t ||^2
  2. 偏好微调模型:     v_{θ^★}(x_t, t, c)    ──> 误差: L_{θ^★} = || v_{θ^★} - u_t ||^2
       │
       ▼
  【原生隐空间过程奖励直出】：
  r_t(x_t, c) = β * [ L_{ref}(x_t, c) - L_{θ^★}(x_t, c) ]
       │
       ├──> 微调模型预测误差更小？ ──> 说明当前隐变量严格符合人类偏好流场！
       ├──> 覆盖连续时间 t ∈ [0, 1] 的所有高噪与低噪节点，彻底告别 VAE 解码；
       └──> 零新增参数，零表征漂移，单卡单模型原生实现生成与评估统一。

                  三大偏好优化算法在流匹配下的统一读出

  算法名称                算法流派              流匹配损失函数 L_F(θ)
  ────────────────────────────────────────────────────────────────────────────────
  RWR (奖励加权回归)      无偏似然流派          E_{t,x_1} [ L_θ(x_w, t) ]
  IPO (恒等边际对比)      非饱和对比流派        E_{t,x_1} [ ( (L_ref - L_θ)_{x_w} - (L_ref - L_θ)_{x_l} - 1/(2τ) )^2 ]
  DPO (Bradley-Terry)     经典饱和对比流派      -E_{t,x_1} [ log σ( β [ (L_ref - L_θ)_{x_w} - (L_ref - L_θ)_{x_l} ] ) ]
  ────────────────────────────────────────────────────────────────────────────────
  三者在收敛极限下，均驱使模型去噪速度差 [L_ref - L_θ] 演变为严格校准的奖励打分器！