Hung-yi Lee ML 2025 Lecture 12: How Language Models Learn to Speak

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

The Vision: Speech Language Models

Imagine an AI that can:

• Listen and speak simultaneously (like humans in conversation)
• Understand tone, emotion, and context from voice
• Generate natural speech with appropriate prosody
• React to non-verbal cues (coughs, laughter, background sounds)

This is the goal of Speech Language Models.

The Challenge: Speech vs Text

Why Speech is Harder

Sequence Length:

• Text: ~100 tokens per minute of speech
• Speech: ~3,000-50,000 tokens per minute (depending on tokenization)

Information Density:

• Text: Just content
• Speech: Content + emotion + speaker identity + prosody + background sounds

Real-time Requirements:

• Text: Turn-based (you type, model responds)
• Speech: Simultaneous listening and speaking

Speech Tokenization: The Foundation

Two Main Approaches

1. Semantic Tokens (from Self-Supervised Models)

• Extracted from models like HuBERT, wav2vec
• Capture content/meaning
• ~12.5-50 tokens per second
• Good for understanding, less for synthesis quality

2. Acoustic Tokens (from Neural Codecs)

• Extracted from models like EnCodec, SoundStream
• Capture audio details (timbre, prosody)
• Multiple “layers” of tokens (coarse to fine)
• Better synthesis quality

The Trade-off

• Semantic tokens: Good understanding, poor audio quality
• Acoustic tokens: Good audio quality, harder to model

Solution: Use both! Semantic for understanding, Acoustic for synthesis.

Architecture Patterns

Pattern 1: Text First, Then Speech

1. Model generates text response
2. TTS converts text to speech

Pros: Leverages existing text LLMs Cons: Can’t do real-time conversation, loses speech-specific info

Pattern 2: Direct Speech-to-Speech

1. Input speech → Speech tokens
2. Language model does token prediction
3. Output tokens → Speech

Pros: Preserves all speech information Cons: Much longer sequences, harder to train

Pattern 3: Hybrid (Moshi-style)

• Generate text AND speech tokens simultaneously
• Text provides semantic guidance
• Speech tokens handle audio details

The Moshi Model

One of the first truly interactive speech language models:

Key Features:

• Simultaneous input/output (full-duplex)
• Generates both text and speech tokens
• Can handle interruptions
• Trained on 7M hours of speech data

Architecture:

• Temporal Transformer: Handles sequence
• Depth Transformer: Generates multiple token layers per step

Training Speech Language Models

Pre-training

• Massive amounts of speech data (100K+ hours)
• Learn speech token prediction (like text LLMs learn word prediction)
• Can use YouTube, podcasts, audiobooks

Fine-tuning for Conversation

• Human-human dialogue data
• Or: Generate synthetic dialogues with text LLM + TTS
• Teach turn-taking, appropriate responses

RLHF for Speech

• Same concept as text RLHF
• But: Need to evaluate speech quality, not just content
• Harder to get human feedback on speech

Key Challenges

1. Sequence Length

A 5-minute conversation might need 500,000 tokens!

• Solutions: Better tokenization, sparse attention, streaming

2. Forgetting

Fine-tuning on speech can destroy text capabilities

• Solutions: Mixed training, careful data curation

3. Real-time Processing

Users expect immediate responses

• Solutions: Streaming generation, speculative decoding

4. Multi-modal Understanding

Understanding coughs, laughter, background noise

• Solutions: More diverse training data, specialized tokens

The Future

What’s Coming

• More natural conversations with AI
• AI that understands emotion and context
• Seamless voice interfaces
• Personalized voice assistants

Open Questions

• How to evaluate speech model quality?
• How to handle multiple speakers?
• How to preserve privacy in voice data?
• Can we reach human-level conversation?

Key Takeaways

1. Speech is not just text + TTS: Real speech models need to understand audio

2. Tokenization matters: The choice of tokens affects everything

3. Sequence length is the bottleneck: Speech needs 10-100x more tokens than text

4. Hybrid approaches work: Combine semantic and acoustic tokens

5. We’re just getting started: Speech LLMs are where text LLMs were 3-4 years ago