Sparser, Faster, Lighter Transformer Language Models

Paper: 2603.23198 Authors: Edoardo Cetin, Stefano Peluchetti, Emilio Castillo, Akira Naruse, Mana Murakami, Llion Jones (Sakana AI, NVIDIA) Categories: cs.LG

Abstract

Scaling autoregressive LLMs has driven unprecedented progress but comes with vast computational costs. This work tackles these costs by leveraging unstructured sparsity within LLM feedforward layers — the components accounting for over two-thirds of model parameters and 80%+ of total FLOPs. The authors introduce TwELL (Tile-wise ELLPACK), a new sparse packing format, and a set of CUDA kernels designed for modern NVIDIA GPUs. Simple L1 regularization induces over 99% sparsity with negligible downstream performance impact. These sparsity levels translate into up to 20.5% forward execution speedup and 21.9% training speedup for billion-parameter models, with benefits increasing at larger scales.

Key Contributions

• TwELL sparse format: A Tile-wise ELLPACK packing format that can be materialized in the epilogue of optimized matrix multiplication kernels, removing a canonical bottleneck of prior sparse schemes
• Fused inference kernels: Multiple matrix multiplications fused into a single optimized pipeline minimizing computation
• Hybrid training format: Reduces sparse representation to trivialize storage costs of intermediate activations
• Sparsity study: Demonstrates that mild L1 regularization achieves 99%+ sparsity with negligible quality loss
• Open-source release: All code and kernels released at github.com/SakanaAI/sparser-faster-llms

The Sparsity Paradox

Despite performing far less theoretical computation, official sparse operation kernels often run slower than dense operations on modern GPUs. The culprit: a fundamental mismatch between unstructured sparsity and GPU architectures, whose hardware and software stacks are heavily optimized for dense computation patterns.

This means that even though LLM feedforward layers naturally exhibit high sparsity (only a small fraction of hidden neurons activate for any given token), converting that theoretical advantage into real-world speedups required solving the systems problem of efficient sparse execution on GPU hardware.

Methodology

TwELL Format

Standard ELL (ELLPACK) format pads all rows to the same length, wasting memory on sparse rows. TwELL groups rows into tiles and stores per-tile metadata, enabling:

• Natural integration with GPU tiling patterns
• Materialization in the epilogue of highly-optimized matmul kernels
• Elimination of the overhead from separate sparsification passes

Inference Kernels

The inference path fuses gating + up-projection + activation + down-projection of feedforward blocks into a single kernel pass, operating directly on sparse intermediate representations.

Training Kernels

For training, a hybrid format combines TwELL with dense representations at different stages, minimizing both compute and the storage cost of activations needed for backpropagation.

Inducing Sparsity

Simple L1 regularization on feedforward hidden activations — no architectural changes, no specialized pruning techniques. The regularization is mild enough that downstream task performance remains essentially unchanged even at 99%+ sparsity.

Results

Metric	Improvement
Forward execution speedup	Up to 20.5%
Training speedup	Up to 21.9%
Sparsity achieved	99%+ with L1 regularization
Performance impact	Negligible
Scaling trend	Benefits increase with model size

Key finding: the benefits come from computational unevenness across network layers and natural language data, which sparse models can inherently leverage — different tokens and different layers exhibit vastly different sparsity patterns.

Takeaways

• Feedforward layers are the dominant cost in LLMs (2/3 of parameters, 80%+ FLOPs) — making them the highest-leverage target for sparsity
• The gap between theoretical sparse FLOPs and real GPU speedup has been a persistent barrier — TwELL and fused kernels close this gap
• 99% sparsity is achievable with simple L1 regularization, no complex pruning needed
• Benefits scale with model size, making this increasingly relevant as LLMs grow
• Open-source release (Sakana AI + NVIDIA collaboration) enables immediate adoption