Do VLMs Actually Need Vision Transformers? A Case for SSM Encoders

Paper: 2603.19209 Authors: Shang-Jui Ray Kuo, Paola Cascante-Bonilla Categories: cs.CV, cs.LG

The Gap

VLMs have a standard recipe: freeze a ViT-based vision encoder (CLIP-ViT, SigLIP, DINOv2), bolt on a lightweight connector, and feed features into an LM. The assumption baked into this recipe is that transformers are the only serious option for the vision side. Nobody really questioned it — partly because ViTs work well, and partly because SSMs (amba-style architectures) were mostly evaluated on language tasks or standalone vision benchmarks, not as frozen backbones inside VLMs.

The specific gap: we don’t know whether SSM vision encoders can hold up in the VLM context, where the backbone is frozen and the downstream tasks include both coarse VQA and fine-grained localization. Prior comparisons were either unfair (different pretraining data, different scales) or incomplete (only VQA, no grounding).

  Prior State                This Paper
  -----------                    ----------
  ViT assumed default            Controlled comparison
  for VLM vision backbone        (matched ImageNet-1K init)
         |                              |
  SSMs only tested on            SSM vs ViT family
  standalone benchmarks          on VQA + grounding
         |                              |
  No dense-task tuning           + detection/segmentation
  ablation for VLMs              adaptation study
         |                              |
  Gap: is ViT actually           Evidence: SSM competitive
  necessary?                     at smaller scale
         |                              v
  Assumption: yes                Conclusion: no, SSM is
                 a strong alternative

The Increment

One sentence: Before this paper, SSM vision encoders were an untested wildcard in VLMs; after it, they’re a validated efficiency-friendly alternative with known failure modes and fixes.

Core Mechanism

The setup is deliberately controlled. Both SSM backbones (specifically VMamba-family) and ViT-family backbones (ViT-B/L, SigLIP, DINOv2) are initialized from ImageNet-1K checkpoints — same data, matched as closely as possible. Each backbone is frozen, then connected to an LLM via a lightweight MLP or cross-attention connector. The whole system is evaluated on VQA benchmarks (VQAv2, GQA, TextVQA) and grounding/localization tasks (RefCOCO, Flickr30k entities).

The second phase introduces dense-task adaptation: before freezing the backbone for VLM training, it gets fine-tuned on detection (COCO) or segmentation (ADE20K). This is motivated by the intuition that localization-heavy VLM tasks might benefit from a backbone that has seen dense supervision. The paper then measures whether this adaptation helps, and by how much, across both families.

The stabilization piece is the most practically interesting. Some backbones — particularly certain ViT variants — show instability on localization tasks (high variance, occasional collapse). The authors identify this and propose simple fixes: gradient clipping, layer normalization adjustments, and careful learning rate scheduling for the connector. These aren’t novel techniques, but applying them diagnostically to backbone-specific failure modes is useful.

  [ImageNet-1K checkpoint]
          |
          v
  +-------+--------+
  |                |
  v                v
[SSM backbone]  [ViT backbone]
  |                |
  | (optional)     | (optional)
  v                v
[Dense-task     [Dense-task
 fine-tune]      fine-tune]
  |                |
  v                v
  +-------+--------+
          |
          v
  [Frozen backbone]
          |
          v
  [Lightweight connector]
          |
          v
  [LLM]
          |
          v
  [VQA / Grounding eval]

Think of this like hiring a photographer for two different jobs: portrait studio work (VQA — coarse, semantic) and sports photography (grounding — fast, precise localization). ViTs are like photographers trained exclusively in studios — great at portraits, but sometimes shaky when the subject moves fast. SSMs are like photographers who trained in both environments. The “dense-task tuning” is like sending your studio photographer to a sports event for a month before the real assignment. The connector is the camera body — it doesn’t matter much which lens you use if the body is good. The key finding is that the SSM photographer gets comparable shots at smaller camera bag (model scale), and the stabilization strategies are just telling the shaky photographer to brace against a wall.

Key Concepts

• State Space Models (SSMs) as vision encoders: SSMs like Mamba process sequences by maintaining a compressed hidden state that gets updated as new tokens arrive — think of it like reading a sentence and keeping a running summary rather than re-reading everything each time (which is what attention does). For images, you flatten the 2D grid into a sequence of patches and run the SSM over them. The upside: linear complexity in sequence length instead of quadratic. The downside historically: SSMs struggle with long-range 2D spatial dependencies because the scan order matters and images don’t have a natural “reading direction.” VMamba addresses this with multi-directional scanning (scan left-to-right, right-to-left, top-to-bottom, bottom-to-top, then merge).

• Frozen backbone in VLMs: When people say the backbone is “frozen,” they mean its weights don’t change during VLM training. Only the connector (and sometimes the LLM) gets updated. This is a deliberate choice — it’s cheaper, and it avoids catastrophic forgetting of the visual representations. But it also means the backbone’s quality is load-bearing: whatever features it learned during pretraining are all you get. This is why the choice of backbone matters so much, and why dense-task tuning before freezing is an interesting lever.

• Grounding/localization as a stress test: VQA is relatively forgiving — you need to understand the image semantically, but you don’t need to know exactly where things are. Grounding tasks (e.g., “find the red chair on the left”) require spatial precision. This is a harder test for vision encoders because it exposes whether the backbone’s spatial representations are actually well-organized. SSMs, with their scan-based processing, might be expected to struggle here — but the paper finds they don’t, which is the surprising result.

Framework Shift

Before (mainstream approach):        After (this paper):

  [ViT encoder]                        [SSM encoder]
       |                                    |
  assumed necessary                    competitive alternative
       |                                    |
  bigger = better                      scale != performance
       |                                    |
  no dense-task tuning                dense-task tuning
  for VLM backbones                    is a useful lever
       |                                    |
  localization instability             stabilization strategies
  unaddressed                          identified + fixed
       |                                    |
  one backbone family                  two families, controlled
  dominates by default                 comparison, honest tradeoffs

From “ViTs are the only serious option” to “SSMs are a validated alternative with known tradeoffs,” the core shift is moving from assumption to evidence.

Expert Assessment

Problem choice: This is a real gap, not a manufactured one. The SSM-for-vision literature (VMamba, Vision Mamba, PlainMamba) has been growing fast, but almost nobody had done a clean aples-to-apples comparison inside the VLM pipeline. The timing is right — SSMs are mature enough to benchmark seriously, and the VLM community is actively looking for efficiency wins. It’s not a groundbreaking question, but it’s a necessary one.

Method maturity: This is a benchmarking paper, not a methods paper, so “clever insight” isn’t really the right frame. The controlled experimental design is the contribution, and it’s done reasonably well. The dense-task tuning ablation is the most interesting methodological choice — it’s not obvious that this would help, and the fact that it does (and roughly equally across families) is a clean finding. The stabilization strategies are pragmatic rather than principled, which is fine for a benchmarking paper.

Experimental integrity: The matched initialization constraint is the right call and the authors are upfront about it. The honest concern is that “matched ImageNet-1K initialization” still leaves a lot of room for confounds — SSM and ViT architectures have different inductive biases, different optimal hyperparameters, and different sensitivity to the connector design. The paper doesn’t fully disentangle these. The localization instability finding is interesting but the sample of affected backbones seems small — it’s hard to know if this is a systematic issue or a few unlucky runs. The stabilization strategies are presented as fixes but the ablation of each strategy individually is thin.

Writing quality: The paper is clear and well-organized, but the discussion section is where it gets lazy. The “higher ImageNet accuracy doesn’t translate to better VLM performance” finding is stated but not really explained — why not? What does this tell us about what VLMs actually need from their vision backbone? That section, if expanded with a proper analysis of which backbone properties correlate with VLM performance, would elevate the whole paper from “useful benchmark” to “actually changes how people think about backbone selection.”

Verdict: weak accept — solid controlled benchmark that fills a real gap, but the analysis stops just short of the insights that would make it genuinely influential.

Takeaways

A few concrete things worth stealing:

1. Dense-task tuning before freezing a backbone for VLM use is a cheap intervention that consistently helps on localization tasks. If you’re building a VLM and care about grounding, run your backbone through a COCO detection fine-tune first. The paper gives you the evidence to justify this to your team.

2. ImageNet accuracy is a bad proxy for VLM backbone quality. Stop using it as a selection criterion. The paper doesn’t tell you what the right proxy is, but it tells you what the wrong one is — which is still useful.

3. If you’re seeing instability in localization tasks with a frozen backbone, the fix is boring but effective: gradient clipping on the connector, check your LR schedule. The paper documents this failure mode clearly enough that you can recognize it when you hit it.

4. For anyone working on efficient VLMs: SSMs at smaller scale are now a credible option. You don’t need to default to a largeViT-L just because that’s what everyone else uses. The efficiency argument for SSMs (linear vs quadratic complexity) is real, and this paper gives you the empirical backing to make the case.