When Rewording the Prompt Flips the Robot's Grade

Paper: 2609.05401 Authors: Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, Yoonjun Cho, Dongjae Jeon, Bumjun Kim, Jean Oh, Youngjae Yu, Albert No Categories: cs.RO, cs.CL

The Gap

Vision-language models (VLMs) are increasingly being promoted from passive perceptual modules to automated evaluators and reward functions for robotic manipulation. In reinforcement learning or trajectory ranking, a VLM reward model takes a video of a robot’s execution and a natural language goal, outputting a scalar progress score or success indicator. For this paradigm to function reliably, it demands a fundamental invariant: paraphrase invariance. If a robot is asked to “Pick radish, then place in pink bowl,” its trajectory should receive the exact same evaluation if the prompt is reworded to “After picking radish, place in pink bowl.”

Prior evaluations of VLM reward models tested accuracy against fixed task strings or simple binary success classifications. They ignored linguistic perturbations. When an agent policy optimizes against a reward model that is sensitive to superficial syntax, the policy does not learn robust motor control; it learns to exploit phrasing artifacts.

This paper establishes the extent of that failure mode. It shows that current state-of-the-art VLMs regularly contradict themselves on identical trajectories under human-verified paraphrases, turning verifiable task completions into failures and vice versa.

[SETTING] Robot Trajectory + Instruction -> VLM Reward Model -> Progress Score
    |
    v
[CURRENT ASSUMPTION] If semantic intent is preserved, reward score is stable
    |
    v
[TEST OF INVARIANCE] Evaluate identical video across verified paraphrases
    |
    +-------------------------+-------------------------+
    v                         v                         v
[Lexical Synonym]         [Syntactic Reorder]       [Action-Goal Inversion]
"pink bowl" -> "magenta"   "Pick X, then Y" ->       "Place X in Y" ->
                           "After picking X, do Y"   "Move X until in Y"
    |                         |                         |
    +-------------------------+-------------------------+
                              v
[OBSERVED REALITY] Contradiction in up to 43% of cases;
identical trajectory scores flip between 1 (failed) and 5 (completed);
RL policies trained on fragile rewards collapse.

The Increment

One sentence: Before this paper, VLM reward models were treated as plug-and-play evaluators with known visual blind spots; after it, they are proven to suffer from severe linguistic fragility, where minor syntactic rewrites invert reward signals on identical physical executions.

Core Mechanism

The authors formalize the vulnerability through RoboRMBench, a benchmark built on 2,390 real-robot trajectories across diverse manipulation tasks, paired with 21,673 human-verified paraphrases. These paraphrases span three operational tiers:

1. Lexical variations: substituting object names, color descriptors, or container attributes with synonyms.
2. Syntactic variations: altering clause structure, such as changing imperative sequences (“Do A then B”) into temporal subordinate clauses (“After doing A, perform B”).
3. Action-goal variations: rewriting procedural commands into goal-state verification queries (“Ensure X is inside Y”).

To diagnose reward models, the paper introduces two primary fragility metrics alongside standard Mean Error (ME):

• Score Contradiction Rate (SCR): The proportion of paraphrase pairs where the reward model assigns divergent scores to the same trajectory that violate basic monotonicity.
• Flip Rate (FR): The percentage of cases where rewording an instruction causes the evaluated outcome to flip between failure (score <= 2) and completion (score >= 4).

   ROBORMBENCH DIAGNOSTIC ARCHITECTURE

  [Real-Robot Video Trajectory: tau]
                 |
        +--------+--------+
        |                 |
        v                 v
   [Prompt g_1]      [Prompt g_2] (Paraphrase)
   "Pick radish..."  "After picking radish..."
        |                 |
        +--------+--------+
                 |
                 v
       [VLM Reward Model R_theta]
                 |
        +--------+--------+
        |                 |
        v                 v
   Score r_1 = 5     Score r_2 = 1  <-- CONTRADICTION!
        |                 |
        +--------+--------+
                 |
                 v
   [Metric Suite: SCR, FR, and Preference Regret]

To explain why this happens, consider a structural metaphor of a courtroom stenographer acting as a judge. The judge is asked to grade a defendant’s compliance with an order. Instead of looking directly at whether the act occurred in the courtroom video footage, the judge parses the grammar of the indictment. If the clerk writes “The subject opened the vault and took the ledger,” the judge awards full marks. If the clerk writes “Prior to taking the ledger, the subject opened the vault,” the judge gets confused by the inverted clause timing, glances at the exact same surveillance tape, and rules that the subject failed to follow the order.

Key Concepts

• Paraphrase Fragility: The vulnerability of multi-modal reward models where superficial syntactic or lexical rewrites of a command induce massive swings in predicted progress scores on identical video input.
• Score Contradiction Rate (SCR): A pairwise metric quantifying how often a model rates trajectory A higher than trajectory B under prompt 1, but rates trajectory B higher than trajectory A under prompt 2.
• Semantic Equivalence Filter: An automated verification pipeline combined with human cross-auditing to guarantee that paraphrased prompts share 100% semantic identity in the robotics domain before benchmark inclusion.

Framework Shift

Before (Standard VLM Reward Evaluation):
[Trajectory] + [Single Fixed Prompt] ---> [VLM Judge] ---> [Reward Scalar]
(Blind to linguistic variance; conflates visual error with prompt sensitivity)

After (RoboRMBench Invariance Protocol):
[Trajectory] + [Paraphrase Set G] ---> [VLM Judge] ---> [Reward Vector]
                                                               |
                                            +------------------+------------------+
                                            v                                     v
                                      [Accuracy (ME)]                     [Invariance (SCR/FR)]

From evaluating a single prompt-response point to evaluating the reward distribution across the semantic equivalence class, the core shift is exposing how reward models hallucinate task failure purely from linguistic variation.

Expert Assessment

Problem choice: Real and urgent. As robotics shifts from hardcoded reward functions to foundation model scoring, uninspected linguistic fragility directly pollutes policy gradients and offline model selection.

Method maturity: The construction of RoboRMBench is rigorous. Collecting 2,390 physical robot executions and verifying 21,673 paraphrases with human cross-checks prevents the benchmark itself from being polluted by semantic drift.

Experimental integrity: The authors test both leading proprietary models (GPT-4o, Gemini 1.5 Pro) and open-source models (Qwen2-VL, InternVL2, LLaVA-OneVision). The findings are consistent across architectures: even frontier models exhibit flip rates exceeding 25% on syntactic reorderings.

Writing quality: The distinction between lexical, syntactic, and action-goal rewrites is cleanly maintained throughout all ablation tables.

Verdict: strong accept — A benchmark that catches a blind spot in robotic foundation models before community bad habits harden into standard practice.

Takeaways

• Never evaluate a robotic policy or reward model against a single phrasing of a goal. Always test over a semantic equivalence cluster.
• When fine-tuning VLMs for reward modeling, syntactic augmentation is just as critical as visual data augmentation.
• In policy deployment, ensemble-averaging reward outputs over 3-5 deterministic syntactic variations acts as an immediate test-time defense against reward hacking.