Elastic Attention Cores for Scalable Vision Transformers

Paper: 2605.12491 Authors: Alan Z. Song, Yinjie Chen, Mu Nan, Rui Zhang, Jiahang Cao, Weijian Mai, Muquan Yu, Hossein Adeli, Deva Ramanan, Michael J. Tarr Categories: cs.CV, cs.LG

The Gap

Vision Transformers dominate computer vision through all-to-all self-attention, where every image patch attends to every other patch. This flexibility comes at a cost: O(N²) complexity that scales quadratically with image resolution. At 1024×1024 resolution, you’re computing attention over 16,384 patches—that’s 268 million pairwise interactions. Prior work assumed this dense connectivity was necessary for learning rich visual representations.

Existing solutions fall into two camps. Sparse attention patterns (Swin, local windows) reduce complexity but sacrifice global context and require architectural gymnastics. Cross-attention approaches (Perceiver, Set Transformers) compress N patches into C bottleneck tokens, achieving O(N) complexity but losing fine-grained spatial information in the bottleneck.

This paper challenges the foundational assumption: do patches really need to talk directly to each other?

Problem: ViT attention scales O(N^2) with resolution
   |
   v
Assumption: Pairwise patch interaction = necessary
   |
   v
Method: Core-periphery structure (patches <-> cores only)
   |
   +---> Evidence: Competitive performance at O(N) complexity
   |
   +---> Conclusion: Direct patch interaction is unnecessary

The Increment

One sentence: Before VECA, you chose between quadratic all-to-all attention or lossy compression through bottlenecks; after VECA, you get linear complexity while maintaining all N patch tokens throughout the network.

Core Mechanism

VECA introduces a small set of C learned “core” tokens (typically 64-256) that act as a communication hub. Image patches never attend to each other directly. Instead, each layer performs two attention operations: patches attend to cores, then cores attend back to patches. The cores are initialized randomly and refined through training, learning to capture the visual patterns most useful for routing information between patches.

Critically, VECA maintains all N input patches throughout the network—no compression, no discarding. The cores don’t replace patches; they mediate communication. Each layer updates both the N patch embeddings and the C core embeddings. Because patches only attend to C cores (not N-1 other patches), complexity drops from O(N²) to O(N·C). Since C is fixed regardless of image resolution, this is effectively O(N).

The architecture supports elastic inference: you can reduce the number of cores at test time to trade accuracy for speed. A model trained with 256 cores can run with 64 cores for faster inference, with graceful performance degradation. This elasticity comes from nested training along the core axis, where the model learns to function with varying core counts.

Standard ViT (O(N^2)):
  Patch1 <---> Patch2 <---> Patch3 <---> ... <---> PatchN
    ^           ^           ^                       ^
    |           |           |                       |
    +-----all-to-all attention (N^2 interactions)--+

VECA (O(N)):
  Patch1   Patch2   Patch3   ...   PatchN
    |        |        |              |
    v        v        v              v
  +---------------------------------------+
  |  Core1   Core2   ...   CoreC         |  <-- learned hubs
  +---------------------------------------+
    |        |        |              |
    v        v        v              v
  Patch1'  Patch2'  Patch3'  ...  PatchN'

  Each patch: 2C interactions (to cores, from cores)
  Total: N * 2C = O(N) for fixed C

Think of VECA like an airport hub system. In a fully connected flight network (standard ViT), every city has direct flights to every other city—that’s N² routes to maintain. VECA instead designates a few hub airports (cores). Cities (patches) only fly to hubs, and hubs fly to cities. You might take two hops to reach your destination, but the total number of routes drops from quadratic to linear. The hubs aren’t destinations themselves—they’re transfer points that learn which connections matter most. And crucially, every city maintains its own airport (all N patches preserved); we’re just changing the route structure, not eliminating cities.

Key Concepts

• Core-periphery attention structure: Instead of a fully connected graph where every node attends to every other node, you have a bipartite structure: periphery nodes (patches) only attend to core nodes, and cores attend back to periphery. This is fundamentally different from bottleneck architectures. In a bottleneck (like Perceiver), you compress N tokens into C tokens and lose information. In core-periphery, you maintain N tokens but route their communication through C mediators. The cores are learned parameters, not compressed versions of the input. They start as random embeddings and learn to capture the communication patterns that matter for the task.

• Elastic inference via nested training: The model is trained to work with variable numbers of cores. During training, you randomly sample different core counts (say, 64, 128, 256) and compute loss for each. The gradient updates teach the model to function across this range. At inference, you can dial down the core count for speed or dial up for accuracy. This is different from typical model compression (pruning, quantization) which requires separate training or fine-tuning. The elasticity is baked into the training process itself.

• Linear complexity without compression: The key insight is that O(N) complexity doesn’t require throwing away information. Standard wisdom says: to avoid O(N²), you must compress N tokens into fewer tokens (bottleneck). VECA shows you can keep all N tokens and still achieve O(N) by changing the attention pattern. Each patch does 2C attention operations (attend to C cores, receive attention from C cores), giving N × 2C = O(N) total operations for fixed C. The patches never directly interact, but information flows between them via the cores.

Framework Shift

Before (all-to-all attention):        After (core-periphery):

[P1]--[P2]--[P3]--[P4]               [P1] [P2] [P3] [P4]
  |  X  |  X  |  X  |                  |    |    |    |
  | X | | X | | X | |                  v    v    v    v
  |X  |X|  X|X|  X|X|                [C1]--[C2]--[C3]
  |  X  |  X  |  X  |                  |    |    |    |
[P5]--[P6]--[P7]--[P8]                 v    v    v    v
                                     [P1] [P2] [P3] [P4]
Dense mesh: N^2 edges
Every patch talks to every           Bipartite: 2*N*C edges
other patch directly                 Patches talk via cores only
                                     All patches preserved

From dense mesh to hub-and-spoke, the core shift is: communication structure, not information capacity.

Expert Assessment

Problem choice: Real gap. High-resolution vision is a genuine bottleneck for ViTs in domains like medical imaging, satellite imagery, and video. The quadratic wall is not manufactured—it’s why practitioners still use CNNs or hybrid architectures for high-res tasks. The paper’s positioning is honest: they’re not claiming ViTs are broken, just that the scaling curve needs fixing.

Method maturity: The core-periphery idea is elegant but not novel in isolation—Set Transformers and Perceiver explored similar territory. What’s new is maintaining all N tokens while achieving linear complexity. The nested training for elasticity is clever and practical. However, the paper doesn’t deeply explore why this works—why can patches communicate effectively through cores without direct interaction? The ablations show it works, but the mechanistic understanding is thin. A simpler baseline to rule out: what if you just used local attention + a few global tokens? That’s also O(N) and might be competitive.

Experimental integrity: Baselines are fair and comprehensive. They compare against Swin, DeiT, and recent efficient transformers across ImageNet classification, ADE20K segmentation, and COCO detection. The numbers hold up—VECA matches or slightly trails state-of-the-art while using less compute. Red flag: the gains are modest. On ImageNet, VECA-B is 0.3% behind DeiT-B at similar FLOPs. The real win is at higher resolutions where quadratic scaling bites, but those experiments are limited. I’d want to see 2048×2048 results to validate the scaling story.

Writing quality: The paper is clear and well-structured. The weakness is in the analysis sections. Section 4.3 (ablations) is thorough but doesn’t build intuition about what the cores are learning. Visualizing core attention patterns or analyzing their semantic roles would elevate the paper from “it works” to “here’s why it works.” The related work section is comprehensive but could be tighter—some citations feel like box-checking rather than positioning.

Verdict: weak accept — Solid contribution with practical value, but the conceptual advance is incremental and the performance gains are modest. The elasticity mechanism is the most interesting piece and deserves more exploration.

Takeaways

The nested training trick for elastic inference is immediately useful beyond this paper. If you’re training any model with a tunable capacity dimension (number of heads, hidden size, etc.), you can train across a range of values simultaneously and get a single model that adapts at inference time. This is cheaper than training multiple models or doing post-hoc compression.

The core-periphery pattern is worth trying in other domains where you have many entities that need to exchange information. Think graph neural networks on large graphs, or sequence models on long documents. The key is identifying when direct pairwise interactions are overkill and a mediated communication structure suffices.

For practitioners hitting the quadratic wall with ViTs: VECA is production-ready. The code is clean, the training is stable, and the performance is competitive. If you’re working with high-resolution images and can’t afford the compute for standard ViTs, this is a viable alternative to Swin or other hierarchical architectures.