NEO-ov: Native One-Vision Models Without Modular Stitching

Paper: 2605.28820 Authors: Haiwen Diao, Jiahao Wang, Penghao Wu, Yuhao Dong, Yuwei Niu, Yue Zhu, Zhongang Cai, Weichen Fan, Linjun Dai, Silei Wu Categories: cs.CV

The Gap

Current vision-language models (VLMs) like LLaVA, Qwen-VL, and GPT-4V follow a modular design: a frozen vision encoder (CLIP, SigLIP) extracts image features, an adapter projects them into language space, and a language decoder generates text. This architecture works well for single images but fragments pixel information across video frames and delays pixel-word interaction until after encoding. Native VLMs (Fuyu, Chameleon) that process raw pixels directly have shown promise on single images but remain unexplored for multi-image scenarios, video understanding, and spatial reasoning tasks.

The boundary: modular VLMs dominate multi-frame tasks but lose fine-grained pixel correspondence. Native VLMs avoid this fragmentation but haven’t scaled to complex temporal scenarios. Can a native architecture compete with modular designs on multi-image and video tasks without sacrificing pixel-level detail?

Problem: Modular VLMs fragment pixels across frames
         Native VLMs unexplored for multi-image/video
                          |
                          v
Assumption: End-to-end pixel-word learning can preserve
            cross-frame correspondence at scale
                          |
                          v
Method: NEO-ov - native transformer processing raw pixels
        + cross-frame attention + unified spatiotemporal modeling
                          |
                          v
Evidence: Competitive with modular VLMs on video/multi-image
          Superior on fine-grained perception tasks
                          |
                          v
Conclusion: Native one-vision architectures are viable at scale

The Increment

One sentence: Before this paper, scaling native VLMs to multi-image and video understanding seemed impractical; after, we have proof that end-to-end pixel-word learning can match modular designs without external encoders.

Core Mechanism

NEO-ov is a single transformer that takes raw pixel patches and text tokens as input, processes them jointly through all layers, and outputs text. No separate vision encoder, no adapter module. Pixels from multiple images or video frames are flattened into patches, concatenated with text tokens, and fed into the transformer. Cross-attention between patches from different frames happens natively within the same attention mechanism that handles text.

The model uses a standard decoder-only transformer architecture. Image patches are embedded via a learned linear projection. Position encodings distinguish spatial location within each frame and temporal order across frames. The transformer layers apply causal masking for text generation while allowing bidirectional attention among image patches. This unified processing means pixel-level features and word embeddings interact from the first layer, not after a frozen encoding stage.

Training follows a three-stage recipe: (1) single-image alignment on image-caption pairs, (2) multi-image instruction tuning with interleaved image-text data, (3) video understanding with temporal data. The model learns to track objects across frames, reason about spatial relationships, and ground language in pixel-level details—all without modular boundaries forcing early commitment to frame-level representations.

Structural metaphor: Think of modular VLMs as a relay race. The vision encoder runs its leg, hands off a baton (image features) to the adapter, which passes it to the language decoder. Each runner specializes but never sees the full track. NEO-ov is a solo marathon runner who sees the entire course from start to finish. Pixels and words are mile markers on the same route, not separate relay legs. The runner (transformer) adjusts pace and strategy continuously based on both visual terrain (pixels) and verbal cues (text) encountered along the way. No handoffs means no information loss at module boundaries, and the runner can backtrack to earlier visual details when later text demands it.

Key Concepts

• Native vision-language modeling: Instead of bolting a language model onto a vision encoder, you build one model that speaks both “languages” from scratch. Imagine learning to read and see simultaneously as a child, rather than learning to see first, then years later learning to read and trying to connect the two. A native model’s neurons can specialize in patterns that span pixels and words—like recognizing that the visual texture of fur correlates with the word “cat”—because it never had to commit to separate visual and linguistic representations. Modular models must learn this correlation indirectly through an adapter bottleneck after visual features are already frozen.

• Cross-frame correspondence: When you watch a video of a ball bouncing, your brain tracks the same ball across frames by matching its appearance, motion, and context. Modular VLMs encode each frame independently, then try to stitch them together afterward—like watching a flipbook where each page is drawn by a different artist. NEO-ov’s attention mechanism can directly compare pixel patch 47 in frame 3 with pixel patch 52 in frame 7, asking “are these the same object?” This direct comparison preserves fine-grained identity across time, crucial for tasks like “which person picked up the red cup in the video?”

• Unified spatiotemporal modeling: Space (where things are in an image) and time (when things happen across frames) are usually handled separately: spatial features extracted per-frame, then temporal features extracted across frames. NEO-ov treats them as one problem. A patch’s position encoding contains both its (x,y) location within a frame and the frame’s temporal index. The transformer’s attention can then learn patterns like “objects in the top-left of frame 5 tend to move toward the bottom-right in frame 6”—spatial and temporal reasoning fused in a single operation, not pipelined through separate modules.

Framework Shift

Before (modular approach):              After (NEO-ov):

Image 1 ---> [Vision Encoder]          Image 1 (raw pixels)
Image 2 ---> [Vision Encoder]          Image 2 (raw pixels)  
Image 3 ---> [Vision Encoder]          Image 3 (raw pixels)  
                  |                            |
            [Feature vectors]            [Patch embeddings]
                  |                            |
              [Adapter]                        |
                  |                            |
         [Language Decoder] <--- Text   [Unified Transformer] <--- Text
                  |                            |
               Output                       Output

Module boundaries = information loss    No boundaries = end-to-end gradients

From pipeline to monolith, the core shift is eliminating handoff points where pixel-level detail gets compressed into frame-level summaries.

Expert Assessment

Problem choice: Real gap. Modular VLMs’ dominance in multi-image/video tasks isn’t because the architecture is optimal—it’s because native models haven’t been seriously attempted at scale. The fragmentation issue (pixels scattered across frames, late pixel-word interaction) is a genuine limitation, not a manufactured problem. This sits at the intersection of two trends: scaling native multimodal models and improving video understanding. Timely and well-motivated.

Method maturity: Mostly straightforward engineering—take a decoder-only transformer, feed it pixels and text, train it carefully. The novelty is in proving it works at scale, not in algorithmic innovation. The three-stage training recipe (single-image → multi-image → video) is sensible but not deeply analyzed. Why this order? What happens if you skip stage 1? The paper doesn’t explore simpler alternatives like hybrid approaches (native for some tasks, modular for others). It’s a brute-force validation that the simple idea works, which is valuable but not intellectually surprising.

Experimental integrity: Baselines are fair—compared against strong modular VLMs (LLaVA-OneVision, Qwen2-VL) on standard benchmarks. The results show NEO-ov is competitive but not dominant: it wins on fine-grained perception tasks (object tracking, spatial reasoning) and loses on some high-level reasoning benchmarks. This is honest reporting. One red flag: the paper claims “largely narrows the gap” but the gap is still 5-10 points on several benchmarks. The architectural analyses (ablations on position encoding, attention patterns) are thorough and useful for future work.

Writing quality: The abstract and introduction oversell the novelty—“native one-vision” sounds more revolutionary than “we removed the vision encoder and it still works.” Section 3 (method) is clear but could be half the length; too much space spent describing standard transformer components. Section 4 (experiments) is well-organized but buries the most interesting finding: native models excel at fine-grained tasks but struggle with abstract reasoning. Rewriting Section 5 (analysis) to deeply explore this tradeoff—why does end-to-end learning help perception but not reasoning?—would elevate the paper from “we built it and it works” to “we understand when and why it works.”

Verdict: weak accept — Solid empirical contribution proving native VLMs scale to multi-image/video tasks, but lacks deep insight into when the architecture is preferable and when modular designs still win.

Takeaways

For practitioners building VLMs: The three-stage training recipe (single-image → multi-image → video) is immediately reusable. Start with abundant image-caption data to learn basic pixel-word alignment, then add multi-image instruction data to learn cross-frame reasoning, finally add video data for temporal dynamics. This curriculum likely transfers to other native architectures.

For video understanding researchers: Native models’ advantage on fine-grained perception tasks (object tracking, spatial grounding) suggests that when your task requires pixel-level correspondence across frames, consider end-to-end architectures over modular pipelines. The tradeoff: you lose the ability to swap in better vision encoders as they’re released.

For architecture designers: The paper’s ablations reveal that position encoding design matters enormously for native models—spatial and temporal encodings must be carefully balanced. Steal this insight: when building models that process structured data (images, graphs, sequences), invest time in position encoding design before scaling up.

Honest assessment: If you’re building a general-purpose VLM for diverse tasks, modular designs still win on flexibility and performance. If you’re targeting fine-grained video understanding (surveillance, robotics, medical imaging), NEO-ov’s approach is worth trying. The paper’s main contribution is proving feasibility, not providing a recipe for dominance.