VISE: Teaching Large Multimodal Models to Actually Look at Images

Paper: 2606.27373 Authors: Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Salman Khan, Fahad Khan Categories: cs.CV

The Gap

Self-evolving large multimodal models (LMMs) use multi-role self-play and self-consistency rewards to improve visual reasoning without labeled data. The problem? These methods optimize answer agreement — that is, they make the model produce the same answer regardless of how the training process is framed. But agreement with itself is not the same as looking at the image. The decoder quickly learns statistical regularities in language (e.g., “a cat sits on a mat” is a safe response for any indoor scene) and uses those priors to generate self-consistent outputs. The model becomes a language guesser that doesn’t bother verifying against visual content. The authors term this failure visual under-conditioning: the attention weights on visual tokens are low, and the decoder effectively bypasses visual evidence. Prior work implicitly assumed that self-consistency would force visual grounding; this paper shows otherwise.

                                    +-------------------+
                                    | Problem: Visual   |
                                    | Under-Conditioning|
                                    | (low attn to img)|
                                    +-------------------+
                                             |
                                             v
+-------------------------+    +----------------------------+
| Assumption: Self-       |    | Reality: Decoder exploits  |
| consistency -> visual   | -> | language priors, ignores   |
| grounding               |    | image. Self-consistency    |
|                         |    | does NOT imply visual      |
+-------------------------+    +----------------------------+
                                             |
                                             v
+------------------------------------------+
| Method: VISE                              |
| -- Geometric Invariance Reward            |
| -- Semantic Invariance Reward             |
| (regularize visual conditioning directly) |
+------------------------------------------+
                                             |
                                             v
+---------------------------+    +----------------------------+
| Evidence: COCO +16.85     |    | Conclusion: Regularizing   |
| CIDEr, TextCaps +19.66,   | -> | visual attention improves |
| Hallucination -5.0 Chair-I|    | LMMs across families      |
+---------------------------+    +----------------------------+

The Increment

One sentence: Before VISE, self-evolving LMMs optimized answer agreement at the cost of visual attention; after VISE, we can directly regularize visual conditioning using unsupervised invariance rewards, achieving large gains across 18 benchmarks.

Core Mechanism

VISE replaces the standard self-consistency reward with two invariance-based signals. Both are computed within a single model — no specialist roles, no external reward models, no annotations. The pipeline: (1) feed a raw unlabeled image through the model’s visual encoder and decoder to get a generated text (e.g., a caption). (2) Apply a known geometric transformation to the image (e.g., rotation, scaling, crop). Feed the transformed image through the same model. The geometric invariance reward penalizes discrepancies between the outputs of the original and transformed images. (3) For semantic invariance: take the model’s output, extract predicted regions (objects or bounding boxes), and perturb (e.g., mask or blur) those regions. Feed the perturbed image through the model. If the model still outputs the same text despite the region being perturbed, it was relying on language priors rather than visual evidence — the reward penalizes that. Both rewards are used to update the model parameters.

+---------+         +----------------+         +-----------+
| Image   | ------> | Visual Encoder | ------> | Decoder   | ------> Text T
+---------+         +----------------+         +-----------+
       |                                              |
       | (a) Geometric Transform                      | (b) Semantic Perturb
       v                                              v
+---------+         +----------------+         +-----------+
| Transf. | ------> | Visual Encoder | ------> | Decoder   | ------> Text T'
| Image   |         +----------------+         +-----------+
+---------+                                        |
                                                   v
+-------------------------------------------------------------+
| Reward Calculation                                          |
| Geom Reward = d(T, T') [low if consistent under transform]  |
| Sem Reward = d(T, T'') [low if consistent under perturbation|
|             + penalty if T still says "there is X" when     |
|             region X is perturbed]                          |
+-------------------------------------------------------------+
                                              |
                                              v
+-------------------------------------------------------------+
| Model Update: minimize combined reward                      |
| (gradient flows through encoder and decoder)                |
+-------------------------------------------------------------+

Here’s a load-bearing metaphor: think of training a security guard to write incident reports. The guard (the decoder) watches a monitor (the visual encoder) and writes a report. Before VISE, the guard is evaluated on whether they write the same report every shift for the same location — consistency. But the guard can be consistent by just writing “All quiet in corridor” every time, never actually looking at the monitor. The geometric invariance reward is like having two monitors at different angles: if the guard’s report changes after switching to the other camera, they weren’t actually watching the scene. The semantic invariance reward is like covering a suspicious package in the video: if the guard still writes “Suspicious package detected” after the package is removed from the frame, they’re clearly hallucinating. VISE penalizes both behaviors, forcing the guard to look at the monitor.

Key Concepts

• Visual under-conditioning: The decoder’s attention mechanism assigns low weight to visual tokens (pixels or patch features) and high weight to textual tokens (language priors). Intuitively, the model “reads the prompt and guesses” instead of “looking at the image and describing.” Concrete example: given an image of a kitchen with no cat, a visually under-conditioned model still outputs “A cat sits on the counter” because that phrase is statistically common in its training corpus for kitchen scenes. The test: if you blank out the image and get the same output, the model is under-conditioned.

• Invariance reward: A training signal that enforces a specific property of the model’s output. Instead of rewarding agreement with a label, VISE rewards: (a) *geometric invariance — output should be similar under common image transformations (rotation, crop) because the semantic content of the scene doesn’t change; (b) semantic invariance — output should change when predicted visual evidence is perturbed, because if the evidence is gone the model should say “I don’t know” or adjust its description. The key insight: invariance is a proxy for visual conditioning. You can’t accidentally satisfy both rewards by just being good at language.

• Self-evolution without self-play: Prior self-evolving LMMs require multi-role self-play (the model plays “teacher” and “student” roles) and self-consistency rewards (rewarding the student for agreeing with the teacher). VISE replaces this entire multi-agent setup with a single model, two forward passes (original + transformed/perturbed), and two simple distance metrics. This is much simpler to implement and directly targets the problem rather than an indirect proxy.

Framework Shift

Before (standard self-evolving LMM):        After (VISE):

[Image] + [Language Prompt]                 [Image] + [Language Prompt]
         |                                            |
         v                                            v
   [Visual Encoder]                             [Visual Encoder]
         |                                            |
         v                                            v
   [Decoder]                                       [Decoder]
         |                                            |
   +-----+-----+                              +-------+-------+
   |           |                              |               |
[Self-Play] [Self-Consistency]          [Geom Transform] [Sem Perturb]
   |           |                              |               |
   +-----+-----+                              +-------+-------+
         |                                            |
         v                                            v
   [Updated Model:                          [Updated Model:
    optimized for                          optimized for
    answer agreement]                      visual conditioning]
         |                                            |
         v                                            v
   [Output: "A cat on mat"                 [Output: "A cat on mat"
    (but maybe no cat in image)]            (only if cat actually there)]

One sentence: From optimizing answer agreement (self-consistency) to optimizing visual conditioning (invariance under geometric and semantic perturbations), the core shift is direct regularization of visual attention via rewards that cannot be gamed by language priors.

Expert Assessment

Problem choice: Real gap. Visual under-conditioning is a known failure mode in LMMs, but prior self-evolving methods overlooked it because they assumed self-consistency implied grounding. This paper cleanly identifies and isolates the issue. Situated at the right spot in the field’s trajectory — after several self-evolving LMM papers but before anyone systemically addressed visual attention in that setup.

Method maturity: Clever insight wrapped in straightforward implementation. The invariance rewards are cheap to compute (two forward passes) and require no external models. However, the method depends on: (1) being able to apply meaningful geometric transformations to the image (requires spatial grounding), and (2) being able to perturb predicted regions (requires the model to output bounding boxes or segmentation). The authors use Qwen3-VL-2B which provides bounding boxes. For models without explicit region prediction, semantic invariance would require a different implementation.

Experimental integrity: Baselines are fair. The paper compares against several state-of-the-art self-evolving LMMs (including prior self-play methods) and reports gains on 18 benchmarks. Gains are large (+16.85 CIDEr on COCO, +19.66 on TextCaps) and consistent across four model families (Qwen, LLaVA, InternVL, and Phi-3-V). The hallucination reduction (-5.0 Chair-I points) is significant. One limitation: the paper does not isolate the contribution of geometric vs. semantic rewards separately in all benchmarks — it lists an ablation but only on COCO. Would like to see on other tasks. Also, all experiments use the same base model (Qwen3-VL-2B) for the main results; generalization across families is shown but with smaller models.

Writing quality: The paper is well-structured but the abstract and introduction could be clearer about *how geometric and semantic rewards are computed. The reader has to dive into Section 3 to get exact formulas. Section 3 itself is dense but complete. The paper would benefit from a single figure showing the two reward branches in one diagram (the current figures are split). A minor quibble: the acronym “VISE” isn’t explained until Section 3, despite being in the title.

Verdict: weak accept — The paper identifies a genuine oversight in self-evolving LMMs, proposes a clean and effective fix, and demonstrates solid empirical gains. Not a breakthrough, but a well-executed contribution that should influence how future self-evolving methods design reward functions.

Takeaways

• Direct visual conditioning reward: The idea of using invariance under geometric transformations and semantic perturbations as a training signal for visual grounding can be ported to any multimodal task that requires visual attention (e.g., VQA with spatial reasoning, referring expression comprehension). If your model is ignoring visual input, try this: pick a transformation that preserves semantics (crop, rotate, color jitter) and penalize output changes.

• Bypassing self-play: One of the most practical takeaways is that you don’t need multi-role self-play to self-evolve. Single-model, multi-pass training with carefully designed invariance rewards is simpler and directly targets the problem. This simplifies engineering and reduces the risk of training instability from role-switching.

• Hallucination diagnostic: The semantic invariance reward (perturb predicted regions and check if output changes) can be used as a *diagnostic for hallucination without needing a separate hallucination detector. If a model’s output doesn’t change after deleting predicted visual evidence, it’s hallucinating. This is a cheap, unsupervised sanity check for deployment.