UIPress: Optical Token Compression for UI-to-Code Generation

Paper: 2604.09442 Authors: Dasen Dai, Shuoqi Li, Ronghao Chen, Huacan Wang, Biao Wu, Qizhen Lan Categories: cs.CL

The Gap

UI-to-Code generation is a token-hungry task: a single screenshot becomes thousands of HTML/CSS tokens. Vision-language models (VLMs) must process massive visual token sequences, creating a latency bottleneck. Prior work tried two approaches, both incomplete. First, inference-time token selection uses task-agnostic heuristics (like attention-based pruning) that don’t adapt to UI structure—they’re generic, not smart. Second, zero-out methods reduce feature magnitude without actually shortening sequences, so prefill latency stays high. Neither truly compresses. Meanwhile, optical compression (learned compression in the encoder) has proven effective for document OCR, but nobody had adapted this paradigm to UI-to-Code. The gap: a learned, task-aware compression method that actually shortens the visual token sequence while preserving UI-critical information.

Problem: VLM processes ~6,700 visual tokens
         |
         v
Assumption: Learned compression can adapt to UI structure
         |
         v
Method: Depthwise-separable convolutions + spatial reweighting
         |
         v
Evidence: 256 tokens match uncompressed quality + 9.1x speedup
         |
         v
Conclusion: Encoder-side compression beats inference-time methods

The Increment

One sentence: Before this paper, UI-to-Code relied on inference-time token selection or zeroing; after, learned encoder-side compression becomes the standard, cutting latency by 9× while improving accuracy.

Core Mechanism

UIPress sits between the frozen ViT encoder and the LLM decoder. It receives ~6,700 visual tokens from the encoder and compresses them to a fixed 256-token budget. The compression pipeline has three stages working in sequence.

First, depthwise-separable convolutions reduce spatial redundancy. These are lightweight convolutions that process each channel independently, then combine results—they’re efficient because they avoid the full cross-channel computation of standard convolutions. Second, element-guided spatial reweighting learns which regions of the UI matter most. The module computes attention-like weights that upweight UI elements (buttons, text, forms) and downweight background noise. Third, a Transformer refinement layer takes the compressed tokens and refines them, allowing cross-token communication to recover lost context. The entire module adds only 21.7M trainable parameters (0.26% of the 8B base model), so it’s a surgical insertion, not a model overhaul.

On the decoder side, Low-Rank Adaptation (LoRA) bridges the representation gap. The compressed tokens have a different distribution than the original 6,700 tokens, so LoRA fine-tunes the LLM’s attention and feed-forward layers to adapt to this new input space without retraining the full model.

ViT Encoder (~6,700 tokens)
         |
         v
Depthwise-Separable Conv (spatial redundancy reduction)
         |
         v
Element-Guided Spatial Reweighting (UI-aware importance)
         |
         v
Transformer Refinement (cross-token context recovery)
         |
         v
256 compressed tokens
         |
         v
LLM Decoder + LoRA (representation adaptation)
         |
         v
HTML/CSS output

Think of UIPress as a museum curator selecting artifacts for a traveling exhibition. The ViT encoder gives you 6,700 artifacts (visual tokens). The curator’s job is to pick 256 that tell the story best. Depthwise-separable convolutions are the curator’s first pass—they group similar artifacts together, removing duplicates. Element-guided spatial reweighting is the curator’s eye for what matters: they know a rare painting (UI button) is more important than a common frame (background). The Transformer refinement is the curator’s final curation—they arrange the 256 selected pieces so they still make narrative sense together. Finally, the museum visitor (LLM decoder) needs to learn how to interpret this curated collection, which is what LoRA does. The visitor’s brain adapts to the new layout, understanding that fewer pieces still convey the full story.

Key Concepts

• Optical Compression: Compression learned during training (encoder-side), not applied at inference time. The model learns which visual information is task-critical and which is noise. Unlike inference-time methods that use fixed heuristics, optical compression adapts to the specific task (UI-to-Code) and dataset. Think of it as training your eye to see what matters, versus squinting at everything equally.

• Depthwise-Separable Convolutions: A factorization trick that splits a standard convolution into two lightweight operations: one per-channel (depthwise) and one cross-channel (pointwise). Standard convolutions are expensive because they compute all channel interactions at once. Depthwise-separable does it in two cheaper steps. For a 256×256 image with 768 channels, standard convolution costs ~150M operations; depthwise-separable costs ~20M. Same output, 7× cheaper.

• Element-Guided Spatial Reweighting: A learned attention mechanism that assigns importance weights to different spatial regions of the UI. A button region gets high weight; empty space gets low weight. This is UI-aware, not generic—it learns that UI elements are information-dense and backgrounds are noise. The module learns these weights during training, so it adapts to the specific UI design patterns in the training data.

Framework Shift

Before (mainstream approach):        After (this paper):
Screenshot                           Screenshot
    |                                    |
    v                                    v
ViT Encoder                          ViT Encoder
    |                                    |
    v                                    v
6,700 visual tokens                  Learned Compression
    |                                    |
    v                                    v
LLM Decoder                          256 visual tokens
    |                                    |
    v                                    v
HTML/CSS                             LLM Decoder + LoRA
                                         |
                                         v
                                     HTML/CSS

Inference-time selection or          Encoder-side learned
zeroing (task-agnostic)              compression (task-aware)
High latency, generic heuristics     Low latency, adaptive

From inference-time token selection to encoder-side learned compression, the core shift is: stop filtering tokens at the last moment; learn to compress them intelligently from the start.

Expert Assessment

Problem choice: This is a real gap. UI-to-Code is a production bottleneck—companies care about latency. Inference-time methods are the current standard, but they’re crude. Optical compression for OCR proved the concept works; adapting it to UI-to-Code is a natural next step. The problem sits at the intersection of practical need and methodological opportunity. Strong framing.

Method maturity: The approach is pragmatic, not flashy. Depthwise-separable convolutions are standard (not novel), spatial reweighting is straightforward attention, Transformer refinement is boilerplate. The novelty is in the combination and the task adaptation, not in individual components. This is actually a strength—the method is reproducible and implementable. No exotic tricks. The LoRA bridge is sensible but feels slightly ad-hoc; a deeper analysis of why the representation gap exists would strengthen this.

Experimental integrity: The comparison is fair—all methods use the same base model (Qwen3-VL-8B). The 9.1× speedup is measured on the same hardware, so it’s not a cherry-picked metric. The CLIP score improvement (+7.5% over uncompressed, +4.6% over the strongest baseline) is solid. One concern: the paper doesn’t report inference-time latency in absolute terms (milliseconds), only relative speedup. Absolute numbers would let readers judge if 9.1× speedup is enough for their use case. Also, the paper doesn’t compare against other learned compression methods (e.g., knowledge distillation, pruning-aware training), only against inference-time baselines. A comparison with other encoder-side methods would strengthen the claim of novelty.

Writing quality: The abstract is dense but clear. The method section is well-structured. One weakness: the paper doesn’t deeply explain *why element-guided spatial reweighting works better than uniform compression. An ablation showing the contribution of each component (convolutions, reweighting, refinement) would help readers understand what’s doing the heavy lifting. The related work section could better position this against optical compression in other domains.

Verdict: Strong accept — This paper solves a real problem with a pragmatic, reproducible method. The 9.1× speedup with quality improvement is compelling. It’s not groundbreaking (components are standard), but it’s the first to combine them for UI-to-Code, and it works. The experimental setup is fair, and the results are convincing. This is the kind of paper that gets deployed.

Takeaways

• Learned compression beats heuristic selection: If you’re building a latency-critical system with large token sequences, consider encoder-side learned compression over inference-time filtering. The task-awareness pays off.

• Spatial reweighting is underrated: For vision tasks with non-uniform information density (UIs, documents, medical images), learning to weight regions differently is cheap and effective. Don’t treat all spatial regions equally.

• Depthwise-separable convolutions are your friend: When you need to compress high-dimensional data (like visual tokens) without blowing up parameter count, depthwise-separable convolutions give you 7-10× efficiency gains. Use them.

• LoRA for representation adaptation: When you compress inputs to a model, the downstream layers see a different distribution. LoRA is a lightweight way to adapt without retraining. This pattern transfers to other compression scenarios.

• Fixed-budget compression is practical: Instead of variable-length compression, fixing the output to 256 tokens makes deployment simpler (no dynamic shapes, easier batching). If you’re compressing for production, consider fixed budgets.