SODA: Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens

Paper: 2602.16687 Authors: Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds, Guangzhi Sun, William Held, Diyi Yang Categories: cs.SD, cs.CL, eess.AS

Abstract

This paper introduces SODA (Scaling Open Discrete Audio), a suite of native audio foundation models that break away from the text-first paradigm dominating current audio language models. Unlike existing approaches that either extend pre-trained text LLM backbones or rely solely on semantic audio tokens, SODA applies next-token prediction directly to audio at scale, jointly modeling semantic content, acoustic details, and text. The research provides three major contributions: systematic investigation of design choices including data sources, text mixture ratios, and token composition; the first scaling law study for discrete audio models through IsoFLOP analysis across 64 models spanning $3 \times 10^{18}$ to $3 \times 10^{20}$ FLOPs; and the training of SODA models ranging from 135M to 4B parameters on 500B tokens, demonstrating their effectiveness as flexible backbones for diverse audio and text tasks.

Key Contributions

• Native Audio Modeling: Introduces a paradigm shift from text-first to audio-native foundation models that jointly process semantic, acoustic, and textual information through unified next-token prediction
• Comprehensive Design Recipe: Systematically investigates and validates design choices for training discrete audio models, including optimal data source combinations, text-to-audio mixture ratios, and token composition strategies
• First Scaling Laws for Audio: Conducts IsoFLOP analysis on 64 models to establish scaling laws for discrete audio models, revealing that optimal data grows $1.6 \times$ faster than optimal model size
• SODA Model Suite: Trains and releases models from 135M to 4B parameters on 500B tokens, demonstrating strong performance across audio generation and cross-modal tasks
• Unified Architecture: Demonstrates versatility through fine-tuning for voice-preserving speech-to-speech translation using the same architecture

Methodology and Architecture

The SODA framework employs a decoder-only transformer architecture that processes interleaved sequences of semantic tokens, acoustic tokens, and text tokens. This design choice enables the model to capture both high-level semantic content and fine-grained acoustic details simultaneously.

Token Representation: The model uses a hierarchical tokenization scheme where audio is represented through multiple token types. Semantic tokens capture linguistic content and meaning, while acoustic tokens encode prosody, speaker characteristics, and other acoustic properties. Text tokens are interleaved to enable cross-modal understanding and generation.

Training Recipe: The systematic investigation reveals several critical insights. The optimal text mixture ratio balances cross-modal capabilities with audio-specific performance. Data source diversity proves essential, combining speech datasets, music, and environmental sounds. The token composition strategy significantly impacts model quality, with interleaved semantic-acoustic sequences outperforming sequential or separate processing.

Scaling Infrastructure: Models are trained using efficient distributed training techniques, with careful attention to computational budget allocation. The training process spans models from 135M parameters (suitable for edge deployment) to 4B parameters (maximizing capability), all trained on a unified 500B token dataset.

Scaling Law Analysis

The scaling law study represents a groundbreaking contribution to understanding how discrete audio models behave as computational resources increase. Using IsoFLOP analysis, the researchers trained 64 models with varying parameter counts and dataset sizes while keeping total compute constant.

Key Findings: The analysis reveals that optimal data size grows approximately $1.6 \times$ faster than optimal model size, a ratio distinct from text-only language models. This suggests that audio models benefit more from data diversity and volume relative to parameter count compared to their text counterparts.

Compute-Optimal Frontier: The study establishes a compute-optimal frontier spanning $3 \times 10^{18}$ to $3 \times 10^{20}$ FLOPs. For a given computational budget, the scaling laws provide precise guidance on how to allocate resources between model parameters and training tokens to achieve optimal performance.

Practical Implications: These scaling laws enable practitioners to make informed decisions about model architecture and training data requirements. For instance, when doubling computational budget, the laws suggest increasing data by approximately 60% and model size by approximately 40% for optimal results.

Validation: The SODA models themselves serve as validation of these scaling predictions, with observed performance closely matching theoretical expectations across the parameter range from 135M to 4B.

Experimental Results and Applications

SODA models demonstrate strong performance across multiple benchmarks and tasks. The evaluation spans general audio generation, speech synthesis, cross-modal understanding, and specialized applications.

Audio Generation Quality: SODA models produce high-fidelity audio with both semantic accuracy and acoustic naturalness. The joint modeling of semantic and acoustic tokens enables the generation of speech that preserves speaker characteristics, prosody, and emotional content while maintaining linguistic correctness.

Cross-Modal Capabilities: By training on interleaved audio and text tokens, SODA develops robust cross-modal understanding. The models can perform audio captioning, text-to-speech synthesis, and audio-conditioned text generation with competitive performance against specialized models.

Voice-Preserving Speech-to-Speech Translation: A key demonstration involves fine-tuning SODA for speech-to-speech translation while preserving the source speaker’s voice characteristics. Using the same unified architecture without task-specific modifications, the model successfully translates speech content while maintaining prosody, speaking style, and voice identity—a challenging task requiring simultaneous understanding of linguistic content, acoustic properties, and cross-lingual mapping.

Efficiency and Scalability: The 135M parameter model provides a compelling efficiency-performance tradeoff suitable for resource-constrained environments, while the 4B parameter model pushes the capability frontier. This range enables deployment across diverse hardware platforms from mobile devices to cloud infrastructure.

Takeaways

1. Native audio foundation models that jointly process semantic, acoustic, and text tokens through next-token prediction offer a more general and flexible approach than text-first architectures
2. Optimal data for discrete audio models grows $1.6 \times$ faster than optimal model size, establishing distinct scaling behavior from text-only language models
3. Systematic design choices—including data source diversity, text mixture ratios around 20-30%, and interleaved token composition—are critical for training effective audio foundation models
4. IsoFLOP analysis across 64 models spanning $3 \times 10^{18}$ to $3 \times 10^{20}$ FLOPs provides actionable scaling laws for resource allocation in audio model training
5. SODA models from 135M to 4B parameters demonstrate that a single unified architecture can serve as a flexible backbone for diverse audio and cross-modal tasks, including challenging applications like voice-preserving speech-to-speech translation