AV-Unified: Teaching One Model to Handle All Audio-Visual Tasks

Paper: 2603.06530 Authors: Guangyao Li, Xin Wang, Wenwu Zhu Categories: cs.CV

The Gap

Audio-visual research has fragmented into specialized silos. You’ve got one model for event localization (finding when a dog barks), another for segmentation (outlining which pixels belong to the barking dog), and yet another for question answering (what color is the dog?). Each task has its own architecture, training pipeline, and dataset format. This isn’t just inconvenient—it’s fundamentally at odds with how humans perceive scenes, where we seamlessly integrate temporal, spatial, and semantic understanding.

The prior approaches—task-specific models like CMBS for segmentation, MUSIC-AVQA for QA, and LLP for parsing—achieve decent performance within their narrow domains. But they can’t share knowledge across tasks. A model that learns to localize dog barks can’t leverage that knowledge to answer “where is the dog?” And critically, they can’t exploit inter-task relationships: knowing when an event happens should inform where it appears spatially.

Problem: Fragmented task-specific models
   |
   v
Assumption: Shared representation can bridge tasks
   |
   v
Method: Token-based unification + multi-scale perception
   |
   v
Evidence: Competitive/superior on 5 benchmarks
   |
   v
Conclusion: Joint learning improves all tasks

The Increment

One sentence: Before, you needed five different models for five audio-visual tasks; now, one model handles them all through a shared token vocabulary.

Core Mechanism

AV-Unified has three main components working in sequence. First, a tokenization layer converts all inputs and outputs—whether they’re temporal boundaries, spatial masks, or text answers—into discrete tokens from a shared vocabulary. This is the key trick: a bounding box becomes a sequence like [x1][y1][x2][y2], a time segment becomes [start][end], and an answer becomes word tokens.

Second, a multi-scale temporal perception module processes the audio-visual input at different time resolutions. Think of it as having multiple clocks running simultaneously—one ticking every second, another every 0.5 seconds, another every 2 seconds. This captures both quick events (a door slam) and slow ones (a conversation). The module uses cross-attention between audio and visual features at each scale, then aggregates them.

Third, a cross-modal spatial perception module handles the spatial dimension. Here’s where it gets clever: visual features guide audio features to learn spatial structure, even though audio doesn’t inherently have pixel-level information. It’s like teaching someone blind to navigate by describing the room layout—the visual modality acts as a teacher for the audio modality through cross-attention and contrastive learning.

Input (video + audio)
   |
   v
+---------------------------+
| Tokenization Layer        |
| (converts to tokens)      |
+---------------------------+
   |
   +---> [Task Prompt] ---+
   |                      |
   v                      v
+---------------------------+
| Multi-scale Temporal      |
| [0.5s] [1s] [2s] clocks  |
| Audio <--> Visual         |
+---------------------------+
   |
   v
+---------------------------+
| Spatial Perception        |
| Visual guides Audio       |
| Cross-attention           |
+---------------------------+
   |
   v
Output tokens --> Decode to task format

Think of AV-Unified as a universal translator at a UN meeting. Different countries (tasks) speak different languages (output formats): one speaks in timestamps, another in pixel coordinates, another in natural language. The translator doesn’t learn each language pair separately. Instead, it converts everything into a common intermediate language (tokens) that captures the essential meaning. The multi-scale temporal module is like having interpreters who specialize in different speaking speeds—some handle rapid-fire debates, others handle slow ceremonial speeches. The spatial perception module is like a visual aide who helps the audio-focused interpreter understand spatial references by pointing at a map. The task prompt is like announcing which country is speaking next, so the translator knows what context to expect.

Key Concepts

• Token Unification: Imagine you’re organizing a messy desk with papers, photos, and sticky notes. Instead of having separate filing systems for each type, you photograph everything and store the photos in one album with labels. That’s what token unification does—it takes diverse data types (bounding boxes, time intervals, text) and converts them into a single format: sequences of discrete symbols. A bounding box (10, 20, 50, 60) becomes tokens [10][20][50][60]. A time segment 2.5s-4.3s becomes [2.5][4.3]. An answer “dog” becomes [dog]. Now your model only needs to learn one language: predicting the next token in a sequence. This isn’t just convenient—it enables weight sharing and transfer learning across tasks.

• Multi-scale Temporal Perception: Audio-visual events happen at wildly different speeds. A gunshot lasts 0.1 seconds. A phone conversation lasts minutes. If you only look at one time resolution, you’ll miss events. It’s like trying to read both a speedometer and an odometer with the same glance—you need different scales. This module runs parallel processing streams at different temporal granularities (say, 0.5s, 1s, 2s windows), then fuses them. Each scale captures different event types: fine-grained for transient sounds, coarse-grained for sustained activities. The fusion happens through learned attention weights that decide which scale matters most for the current input.

• Cross-modal Spatial Guidance: Audio doesn’t come with pixel coordinates, but visual data does. The problem: how do you train a model to localize sound sources spatially when your audio features have no inherent spatial structure? The solution: use visual features as a teacher. During training, the model learns to align audio features with visual features that *do have spatial information. It’s like learning to navigate by sound after someone first shows you a map. The visual modality provides supervision through contrastive learning—audio features corresponding to the same spatial region should be similar, different regions should be dissimilar. This cross-modal guidance lets the audio branch learn spatial awareness indirectly.

Framework Shift

Before (mainstream approach):        After (this paper):

Task 1: Event Localization          +----------------------+
  [Model A] --> timestamps          |   AV-Unified Model   |
                                    |                      |
Task 2: Segmentation                |  [Token Vocab]       |
  [Model B] --> pixel masks         |  [Multi-scale Time]  |
                                    |  [Spatial Guidance]  |
Task 3: Question Answering          |  [Task Prompts]      |
  [Model C] --> text answers        +----------------------+
                                              |
Task 4: Parsing                               v
  [Model D] --> event labels         All outputs as tokens
                                     (timestamps, masks, text)
Task 5: Segmentation                          |
  [Model E] --> audio masks                   v
                                    Decode to task format
No shared knowledge

From isolated specialists to a single polyglot, the core shift is representation unification enabling knowledge transfer.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The field has genuinely fragmented into task-specific solutions, and there’s clear value in unification—both for practical deployment (one model vs. five) and scientific understanding (what’s shared across tasks?). It sits at a natural inflection point where enough individual tasks have matured that unification becomes feasible and valuable.

Method maturity: The token unification idea is borrowed from NLP (think T5, GPT), which is smart but not novel. The multi-scale temporal module is standard practice in video understanding. The genuinely clever bit is the cross-modal spatial guidance—using visual features to teach audio features about spatial structure is an elegant solution to a real problem. However, I’m skeptical about whether this is simpler than just having task-specific heads on a shared backbone. The tokenization adds complexity (quantization, vocabulary design) that might not be necessary.

Experimental integrity: The baselines are fair—they compare against published task-specific models on standard benchmarks. The numbers look solid across five datasets (AVE, LLP, MUSIC-AVQA, VGG-SS, AVS). But there’s a red flag: no ablation showing whether joint training actually helps each individual task, or if it’s just competitive with training separately. The paper claims “joint learning improves all tasks” but doesn’t show single-task baselines trained with the same architecture. Also, the computational cost comparison is missing—does unification save FLOPs or just consolidate models?

Writing quality: The abstract oversells (“naturally integrate multiple audio-visual tasks”) without acknowledging that humans don’t actually tokenize the world. The method section is dense and would benefit from a clearer separation of “what” (architecture) vs. “why” (design choices). The related work section is thorough but reads like a literature dump rather than a narrative. If I could rewrite one section, it’d be the introduction—frame the problem as “knowledge transfer across tasks” rather than “humans integrate tasks,” which is a weak motivation.

Verdict: weak accept — Solid engineering that advances the field incrementally, but the core ideas aren’t novel enough for a strong accept, and the experimental story has gaps.

Takeaways

The big steal here is token-based task unification as a design pattern. If you’re working on any multi-task problem where outputs have different formats (detection boxes, segmentation masks, classifications, text), consider converting everything to tokens. This isn’t just about convenience—it forces you to think about what’s shared across tasks at the representation level.

The cross-modal guidance technique is worth lifting: if you have two modalities where one has richer supervision (e.g., visual has spatial labels, audio doesn’t), use contrastive learning to let the rich modality teach the poor one. This transfers beyond audio-visual to any asymmetric multi-modal setup.

Finally, the multi-scale temporal processing is a reminder that temporal granularity matters. Don’t assume one time resolution fits all events. Run parallel streams at different scales and fuse them. This applies to any temporal modeling problem, not just audio-visual.

What you can’t steal: the specific architecture details are too tied to this problem. The task prompts are a nice touch but not groundbreaking. And the paper doesn’t provide enough insight into when unification helps vs. hurts to guide your own design choices.