Can Language Models Beat FLAC? Benchmarking Neural Compression for Full-Fidelity Audio

Paper: 2603.08683 Authors: Phillip Long, Zachary Novack, Chris Donahue Categories: cs.SD, cs.AI, cs.LG, eess.AS

The Gap

Prior work showed that autoregressive language models trained on raw audio waveforms can compress audio losslessly, but these experiments stopped at 8-bit audio—the quality of a 1980s telephone. Professional audio lives at 16-bit (CD quality) or 24-bit (studio quality), and nobody knew if neural compression could scale there. The vocabulary explosion is the killer: 8-bit needs 256 tokens, 16-bit needs 65,536, and 24-bit needs 16.7 million. Training a language model with a 16-million-token vocabulary is computationally absurd.

Problem: LM-based compression stuck at 8-bit
   |
   v
Assumption: Sample-level tokenization (one token per sample value)
   |
   v
Method: Byte-level tokenization (Trilobyte) - O(1) vocab instead of O(2^b)
   |
   v
Evidence: Benchmark across 8/16/24-bit, multiple domains, beats FLAC at 8/16-bit
   |
   v
Conclusion: LMs work for professional audio, but gains diminish at higher bit depths

The Increment

One sentence: Before this paper, neural audio compression was a toy demo on 8-bit audio; after, it’s a viable alternative to FLAC for professional 16/24-bit recordings with state-of-the-art compression ratios.

Core Mechanism

The method treats audio compression as next-token prediction, but instead of predicting entire sample values (which would require millions of tokens for 24-bit), it predicts bytes. A 24-bit audio sample is three bytes. The model sees the first byte, predicts the second, then sees both and predicts the third. This sequential byte prediction turns an intractable vocabulary problem into a manageable one—you only need 256 tokens regardless of bit depth.

The architecture is a standard autoregressive transformer. Audio samples flow in as byte sequences, the model builds context across time, and outputs probability distributions over the next byte. During compression, you encode the actual byte using arithmetic coding based on the model’s predicted probabilities. During decompression, you sample from those probabilities to reconstruct the original bytes perfectly.

Audio Sample (24-bit):  [byte1][byte2][byte3]
                           |      |      |
                           v      v      v
Transformer:          context -> P(byte2|byte1) -> P(byte3|byte1,byte2)
                           |      |      |
                           v      v      v
Arithmetic Coder:      encode actual bytes using predicted probabilities
                           |
                           v
Compressed Bitstream:  [bits representing byte sequence]

Think of this like a translator working with an unfamiliar alphabet. Instead of memorizing 16 million unique symbols (sample-level), they learn 256 letters (byte-level) and spell out each word letter-by-letter. A 24-bit sample is a three-letter word. The translator reads the first letter, guesses the second based on patterns they’ve learned, then reads both and guesses the third. If their guesses are good (high probability on the correct letter), the word compresses well. The arithmetic coder is the shorthand system that writes down only the surprising parts—if the translator was 99% sure about a letter, you barely need any ink to record it.

Key Concepts

• Arithmetic Coding: Imagine you’re encoding a message where some letters are very predictable and others aren’t. Instead of using fixed-length codes (like ASCII’s 8 bits per character), arithmetic coding uses fewer bits for probable outcomes and more for improbable ones. If your model predicts the next byte is 0x42 with 90% confidence and it actually is 0x42, you only need about 0.15 bits to encode it (log₂(1/0.9)). If the model is surprised (predicted 1% probability), you need about 6.6 bits. The better your predictions, the smaller your compressed file. It’s the mathematical formalization of “don’t waste space stating the obvious.”

• Byte-Level Tokenization: Standard language models treat each possible value as a distinct token—for 24-bit audio, that’s 16.7 million tokens, each needing its own embedding vector and output weight. Byte-level tokenization breaks each sample into its constituent bytes (3 bytes for 24-bit) and predicts them sequentially. The vocabulary stays at 256 tokens regardless of bit depth. The tradeoff: you predict 3 times per sample instead of once, but your model is 65,000x smaller in vocabulary size. It’s like switching from a dictionary with every possible three-letter combination to one with just the 26 letters of the alphabet.

• Lossless Compression: Unlike MP3 or AAC which throw away information humans supposedly can’t hear, lossless compression preserves every bit of the original audio. You can decompress and get bit-for-bit identical data. The challenge is that audio has complex structure—nearby samples are correlated, but the patterns vary wildly between a symphony, a podcast, and a bird call. Traditional codecs like FLAC use hand-crafted predictors and entropy coding. Language models learn the patterns from data, potentially capturing structure that hand-crafted rules miss.

Framework Shift

Before (FLAC/traditional):           After (Trilobyte/LM-based):

Audio -> Hand-crafted -> Entropy     Audio -> Byte -> Transformer -> Arithmetic
         Predictor       Coder                Split   (learned       Coder
         (linear)        (fixed)                      predictor)     (adaptive)
            |               |                            |               |
            v               v                            v               v
      Fixed rules      Compress                   Learned patterns  Compress
      (domain-agnostic)                          (domain-adaptive)

From hand-crafted prediction rules to learned prediction models, the core shift is replacing domain-agnostic linear predictors with domain-adaptive neural networks that discover compression-relevant patterns from data.

Expert Assessment

Problem choice: This is a real gap with practical stakes. The 8-bit limitation in prior work made neural audio compression a curiosity, not a tool. Professional audio is 16/24-bit, and showing whether neural methods scale there matters for both ML research and audio engineering. The problem sits at the intersection of two mature fields (compression and language modeling), which is often where interesting things happen.

Method maturity: Trilobyte is clever but not revolutionary—byte-level tokenization is borrowed from text LMs handling Unicode. The insight is recognizing that audio samples are just multi-byte integers, so the same trick applies. The execution is solid: they benchmark properly across domains, bit depths, and sampling rates. No simpler approach is being overlooked; this is the natural next step after prior 8-bit work hit the vocabulary wall.

Experimental integrity: Baselines are fair—they compare against FLAC, the industry standard, and include ablations on tokenization schemes. The numbers are credible: LMs beat FLAC at 8/16-bit but gains shrink at 24-bit, which makes sense (higher bit depth means less redundancy to exploit). One minor flag: they don’t report training costs or inference speed, which matters for practical adoption. Compression ratio alone doesn’t tell you if this is deployable.

Writing quality: The paper is clear and well-structured. The weakness is in the discussion section—they observe diminishing returns at higher bit depths but don’t deeply explore why. Is it fundamental (less structure to model) or contingent (model capacity, training data)? A paragraph of speculation with follow-up experiments would strengthen the contribution. The related work section is thorough but could be trimmed.

Verdict: weak accept — Solid empirical work that answers an open question (do LMs scale to professional audio?) with a practical solution (byte-level tokenization), but the contribution is incremental rather than paradigm-shifting.

Takeaways

The byte-level tokenization trick transfers directly to any domain with multi-byte data: high-resolution images, scientific sensor data, genomic sequences. If you’re hitting vocabulary explosion with sample-level tokenization, break samples into bytes and predict sequentially. The tradeoff is always the same: smaller vocabulary, more prediction steps, but tractable training.

The diminishing returns at higher bit depths is a useful negative result. It suggests neural compression’s advantage comes from modeling structure in the high-order bits (which carry most of the signal), not from squeezing out the last bit of entropy in the low-order bits (which are nearly random). If you’re working on compression for high-precision data, focus your model capacity on the structured parts.

The benchmark itself is valuable infrastructure. They tested across music, speech, and bioacoustics at multiple sampling rates and bit depths. If you’re developing a new compression method, this gives you a standardized evaluation protocol to compare against.