Audio-Visual Intelligence: A Survey of Foundation Models Bridging Sound and Sight

Paper: 2605.04045 Authors: You Qin, Kai Liu, Shengqiong Wu, Kai Wang, Shijian Deng, Yapeng Tian, Junbin Xiao, Yazhou Xing, Yinghao Ma, Bobo Li Categories: cs.CV

The Gap

Audio-visual AI has exploded in the foundation model era—Meta MovieGen generates synchronized video with sound, Google Veo-3 creates cinematic clips from text. But the field is a mess. Researchers use inconsistent taxonomies (is audio-driven video generation “understanding” or “generation”?), evaluate on incomparable benchmarks, and publish methods that can’t be systematically compared. Prior surveys either predate the foundation model shift or focus narrowly on single tasks like video captioning or sound localization. No one has mapped the full landscape—understanding, generation, and interaction—through the unified lens of how large models actually work: tokenization, cross-modal fusion, autoregressive vs diffusion architectures, instruction tuning.

Fragmented Literature          Unified Taxonomy
(inconsistent tasks)    -->    (understand/generate/interact)
        +                               +
Heterogeneous Methods   -->    Shared Foundations
(can't compare)                (tokenization, fusion, training)
        +                               +
Scattered Benchmarks    -->    Structured Evaluation
(no integration)               (datasets, metrics, challenges)
        ||
        v
   This Survey: First comprehensive framework for AVI in foundation model era

The Increment

One sentence: Before this paper, audio-visual AI was a collection of disconnected subfields; after, we have a unified map showing how foundation models handle the full spectrum from speech recognition to video-to-audio synthesis.

Core Mechanism

This isn’t a methods paper—it’s a survey that organizes the chaos. The authors establish a three-tier taxonomy. Tier 1 splits tasks into understanding (perceive multimodal input), generation (create audio or video from the other modality), and interaction (dialogue, embodied agents). Tier 2 breaks each into subtasks: understanding includes speech recognition, sound localization, audio-visual segmentation; generation covers text-to-video with audio, audio-driven talking heads, video-to-audio; interaction spans conversational agents and embodied systems. Tier 3 dives into methodological foundations shared across tasks: how to tokenize continuous audio/video into discrete tokens, how to fuse modalities (early fusion, late fusion, cross-attention), whether to use autoregressive models (like GPT) or diffusion models (like Stable Diffusion), and how to scale pretraining and align models with human preferences.

                    Audio-Visual Intelligence
                            |
        +-------------------+-------------------+
        |                   |                   |
   Understanding        Generation         Interaction
        |                   |                   |
   [speech recog]      [text->video+audio]  [dialogue agents]
   [sound local]       [audio->video]       [embodied AI]
   [AV segment]        [video->audio]       [agentic systems]
        |                   |                   |
        +-------------------+-------------------+
                            |
                  Shared Foundations
                            |
        [Tokenization] -> [Fusion] -> [Architecture] -> [Training]
         (audio/video)   (cross-modal)  (AR/diffusion)  (pretrain/align)

Think of this survey as building a subway map for a city that grew organically. Audio-visual AI is like a sprawling metropolis where neighborhoods (tasks) developed independently—some have bullet trains (mature methods), others dirt roads (nascent research). Researchers in the “speech recognition” district rarely visit the “video-to-audio” quarter, and when they do, they can’t find their way because street signs use different languages. This survey draws the first complete metro map. It identifies the major lines (understanding, generation, interaction), marks all the stations (specific tasks), and—crucially—shows the transfer hubs (shared methodological foundations) where techniques from one line can transfer to another. The tokenization hub connects discrete NLP methods to continuous audio/video. The fusion hub shows how to merge modalities. The architecture hub reveals that autoregressive and diffusion models are two rail systems serving the same city. Now a researcher can see: “I’m working on audio-driven video, but the instruction tuning techniques from conversational agents might apply here.”

Key Concepts

• Modality Tokenization: Audio and video are continuous signals—waveforms and pixel arrays that change smoothly over time. Foundation models like GPT or DALL-E work with discrete tokens (words, image patches). How do you bridge this? For audio, you can use codecs (like EnCodec) that compress waveforms into sequences of integers, or learned embeddings from models like HuBERT. For video, you split frames into patches (like ViT does for images) or use 3D convolutions to capture spatiotemporal chunks. The key insight: tokenization isn’t just compression—it’s choosing what level of abstraction the model sees. Coarse tokens (fewer codes) lose detail but are easier to model; fine tokens preserve nuance but explode sequence length. Example: MovieGen uses a hierarchical codec with 12 levels—coarse tokens capture rhythm and pitch, fine tokens capture timbre and texture.

• Cross-Modal Fusion: You have audio tokens and video tokens. How do you make them talk to each other? Early fusion concatenates them before processing (like stacking audio and video frames into a single tensor). Simple but rigid—the model must learn alignment from scratch. Late fusion processes each modality separately, then combines high-level features (like averaging embeddings). Flexible but misses low-level correlations (e.g., lip sync). Cross-attention is the sweet spot: audio tokens attend to video tokens and vice versa, letting the model learn which parts of one modality correspond to which parts of the other. Example: In audio-driven talking heads, cross-attention lets the model focus on phoneme tokens when generating mouth shapes, and prosody tokens when generating head motion.

• Autoregressive vs Diffusion: Two paradigms for generation. Autoregressive models (like GPT) generate one token at a time, left to right, conditioning each token on all previous ones. Fast to sample, easy to control (you can steer generation mid-stream), but error accumulation—one bad token poisons the rest. Diffusion models (like Stable Diffusion) start with noise and iteratively denoise, refining the entire output in parallel. Slower (requires many denoising steps) but higher quality and better at capturing global structure (e.g., long-range audio-video synchronization). The field is split: autoregressive dominates text-to-video (because it’s easier to scale to long sequences), diffusion dominates video-to-audio (because audio quality matters more than speed). Some recent models (like Transfusion) try to combine both.

Framework Shift

Before (task-specific silos):        After (unified foundation model view):

Speech Recognition                   Understanding Tasks
    [custom ASR model]                   |
                                     [shared tokenizer]
Sound Localization                       |
    [custom detector]                [cross-modal fusion]
                                         |
Text-to-Video                        [foundation backbone]
    [custom T2V model]                   |
                                     [task-specific heads]
Video-to-Audio                           |
    [custom V2A model]               Generation Tasks
                                         |
Dialogue Agents                      Interaction Tasks
    [custom chatbot]

Each task reinvents the wheel.       Shared infrastructure, specialized outputs.
No knowledge transfer.               Techniques transfer across tasks.

From isolated task-specific models to a layered architecture where tasks share tokenization, fusion, and backbone, differing only in output heads and training objectives.

Expert Assessment

Problem choice: This is real infrastructure work. The field genuinely needed a map—I’ve reviewed papers where authors cite “audio-visual learning” but mean completely different things. The timing is right: foundation models have matured enough that patterns are emerging, but not so mature that this is obvious. The gap is legitimate.

Method maturity: Not applicable—this is a survey, not a methods paper. But the taxonomy is well-designed. The three-tier structure (task families → specific tasks → shared foundations) is more useful than prior surveys’ flat lists. The decision to organize around foundation model components (tokenization, fusion, architecture) rather than application domains is the right call for 2025+.

Experimental integrity: Surveys don’t have experiments, but the authors curate datasets and benchmarks across tasks. The comparison tables are dense but fair—they don’t cherry-pick metrics to favor certain methods. One weakness: the “open challenges” section (synchronization, spatial reasoning, controllability, safety) is generic. Every multimodal survey lists these. I wanted more specificity: *which synchronization problems are actually hard? Lip sync is mostly solved; what about subtle audio-visual correlations like footstep timing or ambient sound propagation?

Writing quality: The paper is comprehensive but exhausting. At 20+ pages, it tries to cover everything and ends up feeling like a reference manual rather than a narrative. The introduction is strong—it motivates the need for unification clearly. But the middle sections (task descriptions, method details) are dense catalogs. The paper would be stronger if the authors picked 3-4 representative tasks, explained them deeply with running examples, then summarized the rest in tables. The “future directions” section is the weakest—it’s a wishlist without prioritization. Which problems are tractable in 2 years vs 10 years?

Verdict: weak accept — Valuable infrastructure work that will be widely cited, but the execution is more encyclopedia than insight. The taxonomy is the contribution; the rest is thorough but not revelatory.

Takeaways

For practitioners building audio-visual systems: Don’t reinvent tokenization. Use established codecs (EnCodec for audio, ViT-style patching for video) unless you have a compelling reason. Cross-attention is the default fusion mechanism—early/late fusion are only justified if you have extreme compute constraints or need real-time performance.

For researchers: The “shared foundations” framing is the steal. If you’re working on a niche audio-visual task, check what techniques from other tasks in the taxonomy might transfer. Example: instruction tuning (from dialogue agents) is underexplored in generation tasks—can you fine-tune a video-to-audio model with natural language instructions like “make the footsteps sound heavier”?

For anyone evaluating foundation models: The survey’s benchmark curation is useful. They identify that many audio-visual benchmarks are too small (< 1000 samples) or too narrow (single domain like music videos). If you’re building a new model, test on diverse benchmarks across task families to avoid overfitting to one distribution.

The meta-lesson: When a field fragments, the first comprehensive map becomes infrastructure. This survey will shape how people think about audio-visual AI for the next 2-3 years, not because it has novel insights, but because it provides a shared vocabulary.