Seeing is Learning: Visual Pretraining Beats Text for Language Models

Paper: 2607.09657 Authors: Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Yunhua Zhou, Demin Song, Kuikun Liu, Haochen Ye, Haian Huang Categories: cs.CV, cs.AI, cs.MM

The Gap

The dominant paradigm in foundation model pretraining has been straightforward: collect massive text corpora, train large language models on them. This works well for prose, but it creates a blind spot. Many real-world knowledge sources — scientific papers, textbooks, web pages, slides — are inherently visual. They contain figures, equations in typeset form, tables with spatial layouts, diagrams with arrows showing relationships, and page structures that encode hierarchy. When current pipelines (OCR tools, HTML-to-text converters, PDF parsers) flatten these into plain text, they destroy information that was never meant to be serialized linearly.

Prior work has acknowledged this in narrow settings — some papers use OCR pipelines with layout awareness, others add visual features as auxiliary inputs. But nobody has systematically asked the simpler, more radical question: what if we just skip text extraction entirely and pretrain on the raw visual appearance of documents? This paper fills that gap by conducting a large-scale, controlled comparison of visual vs. text pretraining on the same underlying corpora, across multiple backbone architectures, with rigorous benchmarking.

                  Problem
                     |
     Documents carry visual knowledge
     (figures, layouts, equations)
                     |
                  Assumption
                     |
     "Convert to text first" is standard
     Visual information is discarded
                     |
                  Method
                     |
     Skip text extraction entirely
     Pretrain directly on visual documents
     Compare: visual vs text-only, same data
                     |
                  Evidence
                     |
     Multiple backbones + benchmarks
     Visual pretraining wins consistently
                     v
                Conclusion
                     |
     Visual pretraining is a scalable,
     superior pathway to language intelligence

The Increment

One sentence: Before this paper, everyone assumed you needed to extract text from documents to train language models; after this paper, we have strong evidence that the visual page itself is a better training signal than the text extracted from it.

Core Mechanism

The approach is conceptually clean. Instead of running documents through an OCR or text-extraction pipeline, the system renders each page as an image. This image — complete with fonts, figures, equations, table formatting, and spatial layout — becomes the raw input for pretraining. The model learns to predict masked or corrupted portions of this visual input in an unsupervised fashion, similar in spirit to masked language modeling but operating over visual tokens.

The authors explore several pretraining paradigms applied to these visual documents: masked image modeling, where patches of the rendered page are masked and the model learns to reconstruct them; contrastive objectives that pull together different views of the same document; and autoregressive visual prediction over spatial regions. The key design principle is that the model never sees extracted text during pretraining — it must learn everything from the visual structure of the page itself. This forces the model to develop representations that encode not just the characters on the page, but their visual context: what equation is nearby, what figure illustrates the point, how the layout organizes the argument.

The scaling study is what makes this convincing rather than just suggestive. They run controlled experiments across different backbone architectures (varying capacity) and different corpus sizes, always keeping the underlying documents identical between visual and text-only conditions. The visual pretraining wins at every scale, and the gap holds or widens as models get larger.

[Visual Pretraining Pipeline]

  Document (PDF, HTML, etc.)
         |
         v
  Render as image  -----> Full page image with
         |                figures, equations,
         |                layout, fonts intact
         v
  Visual tokenizer
         |
         v
  Patch / grid tokens
         |
         +-----> Masked visual modeling
         |       (predict masked patches)
         |
         +-----> Contrastive learning
         |       (same doc, different views)
         |
         +-----> Autoregressive visual
                 (predict next spatial region)
         |
         v
  Pretrained visual-language model
  (ready for downstream tasks)

Structural metaphor: Think of learning to cook from a recipe book versus learning by watching a master chef work in the kitchen. The recipe book (text extraction) tells you “add two tablespoons of butter” and “stir until golden.” But when you watch the chef (visual pretraining), you see *how the butter melts, what “golden” actually looks like, how the pan is tilted, what the texture is at each stage. You pick up spatial relationships, timing cues, and visual patterns that the recipe text never captured. The recipe didn’t lose information on purpose — it just couldn’t encode it. Similarly, text extraction from a scientific paper doesn’t deliberately throw away the bar chart or the chemical diagram — it just has no way to serialize that spatial, relational information into a linear sequence of tokens. The visual pretraining model is the chef’s apprentice who learns by watching, not reading.

Key Concepts

• Visual tokenization of documents: Imagine taking a photograph of a page from a textbook. You then chop that photograph into a grid of small squares — maybe 16x16 pixels each. Each square becomes a “token,” just like a word is a token in text. Now the model sees a sequence of visual tokens instead of word tokens. The crucial difference: a visual token from the middle of a graph encodes spatial position, color, shape, and context all at once, whereas the word “Figure 3” in extracted text tells you almost nothing about what the figure actually contains. This is the foundational move — treating the rendered page as a first-class input modality, not a preprocessing artifact.

• Information loss in text extraction: Take a concrete example. A chemistry paper contains a structural diagram of a molecule with bonds drawn as lines between atoms arranged in a hexagonal ring. An OCR pipeline might output “benzene ring” or even just garbled coordinates. A human reader looking at the original diagram instantly grasps the symmetry, the bond angles, the spatial relationships. The paper’s central claim is that this gap between what’s in the visual document and what survives text extraction is not a minor nuisance — it’s a systematic source of knowledge loss that scales with the complexity of the source material.

• Same-corpus controlled comparison: This is the experimental design that makes the results trustworthy. It’s not that the visual pretraining model was trained on *more data or better data. It was trained on the same documents — the only difference is that one model saw the rendered pages and the other saw the extracted text. This controls for data quality, data scale, and data source, isolating the effect of the representation format itself.

Framework Shift

Before (mainstream approach):        After (this paper):
                                     
  Raw documents                      Raw documents
       |                                  |
       v                                  v
  Text extraction                  Render as images
  (OCR, HTML parse)                (keep visual fidelity)
       |                                  |
       v                                  v
  Plain text corpus                Visual document corpus
       |                                  |
       v                                  v
  Train LM on text tokens          Train model on visual tokens
       |                                  |
       v                                  v
  Language model                   Visual-language model
  (lost visual info)               (preserved visual info)

From treating text extraction as an unquestioned preprocessing step to treating the visual page as a superior training signal, the core shift is recognizing that the format of your training data is itself a design choice with measurable consequences.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The assumption that “documents must be converted to text to train language models” has been so deeply ingrained that few people even questioned it. The paper correctly identifies that this assumption discards information systematically, not occasionally. It sits at the intersection of multimodal learning and NLP pretraining — a space that’s been mostly about vision-language alignment (e.g., CLIP-style models) rather than asking whether vision could *replace text as the pretraining modality for language tasks. Good instinct.

Method maturity: The method is more brute force than elegant. The core idea — just render and train on pixels — is almost aggressively simple. There’s nothing architecturally novel here; the contribution is the empirical finding, not a clever algorithm. One could argue that more sophisticated visual tokenization (learned patches, hierarchical representations) could improve things further, but the paper doesn’t explore this deeply. The simplicity is both a strength (easy to reproduce, clean comparison) and a weakness (we don’t know where the gains come from — is it the figures? the layout? the equations?).

Experimental integrity: The controlled comparison design is solid — same corpus, different representations, multiple backbones. This is the right way to isolate the variable. The benchmarks appear diverse enough to be convincing. One concern: the paper would benefit from more ablation on *which visual features drive the gains. Is it figures? Equations? Layout structure? Without this, the result is convincing but not fully actionable. Also, computational cost comparisons would strengthen the case — visual tokens are typically more expensive than text tokens, so “better at same scale” might hide a cost penalty.

Writing quality: The abstract is clean and the motivation is well-articulated. However, the paper could be sharper in its related work section — there’s a long line of document understanding work (LayoutLM, DocFormer, etc.) that uses visual features for downstream tasks. The paper needs to more crisply distinguish “visual features help on document tasks” (known) from “visual pretraining is better than text pretraining for general language intelligence” (novel claim). The discussion section, if rewritten to address computational tradeoffs and failure modes, would significantly elevate the work.

Verdict: weak accept — The controlled comparison is rigorous and the finding is surprising enough to matter, but the lack of mechanistic understanding (why does it work?) and computational analysis (at what cost?) keeps it from a strong accept.

Takeaways

Three things you can steal:

1. Question your preprocessing pipeline: If your pipeline converts rich data to a simpler format before training, you may be losing more than you think. This applies beyond documents — any domain where source data has spatial, structural, or visual richness (medical images with annotations, architectural blueprints, music scores) could benefit from a similar audit.

2. Same-corpus controlled comparison as a methodology: The experimental design itself is reusable. Whenever you want to argue that representation X is better than representation Y for training, keep the data identical and change only the representation. This sounds obvious but is rarely done this cleanly.

3. The “render, don’t extract” paradigm for document-heavy NLP: If you’re working with scientific papers, legal documents, or any visually rich text source, consider rendering pages as images for pretraining rather than relying on text extraction. Even if you ultimately need text for downstream tasks, the pretraining representations may be richer if they were learned visually.