FlatSounds: Testing Whether Video-to-Audio Models Understand Physics or Just Fake It

Paper: 2605.30339 Authors: Tingle Li, Siddharth Gururani, Kevin J. Shih, Gantavya Bhatt, Sang-gil Lee, Zhifeng Kong, Arushi Goel, Gopala Anumanchipalli, Ming-Yu Liu Categories: cs.CV, cs.MM, cs.SD, eess.AS

The Gap

Video-to-audio (V2A) models have reached impressive perceptual quality—generated soundtracks sound plausible to human ears. But existing evaluations only measure whether the audio sounds good, not whether it reflects the actual physics in the video. Prior work uses metrics like Fréchet Audio Distance (FAD) and CLAP scores that reward perceptual realism but ignore physical correctness. No one has systematically tested: if you change one physical variable in a video (object size, drop height, material), does the generated audio change accordingly?

The gap: we can’t distinguish models that understand physics from models that memorize plausible audio patterns. This matters because physical reasoning is the foundation for controllable, interpretable generation—without it, V2A models are sophisticated pattern matchers, not physics simulators.

Problem: V2A models sound good but physical grounding unknown
    |
    v
Assumption: Physical correctness != perceptual realism
    |
    v
Method: Controlled counterfactual pairs + single-video pattern tests
    |
    v
Evidence: Models rely on captions > pixels for physics
    |         Captions improve physics but hurt temporal alignment
    v
Conclusion: Need to learn physics from pixels, not text shortcuts

The Increment

One sentence: Before this paper, we evaluated V2A models on whether audio sounds plausible; after, we can measure whether they capture the physics that *causes the sound.

Core Mechanism

FlatSounds constructs two types of tests. First, counterfactual pairs: take a video (e.g., ball drop), vary one physical factor (drop height: 1m vs 2m), generate audio for both, and check if the audio difference matches the physical difference (higher drop → louder impact). Second, single-video pattern tests: within one video, probe whether the model maintains internal consistency (same object → same timbre) and follows directional trends (object moves closer → sound gets louder).

The benchmark covers four physical factors: object size, drop height, material properties, and impact timing. Each factor has ground-truth physical relationships (larger object → lower pitch, harder material → sharper attack). The authors generate videos with controlled variations, feed them to V2A models with and without text captions, then measure whether the generated audio respects these relationships using specialized metrics (pitch deviation, loudness curves, spectral features, temporal alignment).

Input Video Stream:
  [Frame 1] [Frame 2] ... [Frame N]
       |         |            |
       +-------- | -----------+
                 v
         V2A Model (with/without caption)
                 |
                 v
         Generated Audio Waveform
                 |
                 v
    +------------+------------+
    |                         |
    v                         v
Physical Metrics         Temporal Metrics
(pitch, loudness,        (onset alignment,
 spectral shape)          sync accuracy)
    |                         |
    +------------+------------+
                 v
    Does audio reflect physics?

Think of FlatSounds as a physics exam for V2A models. Existing benchmarks are like judging a student’s essay on writing style alone—does it sound eloquent? FlatSounds is the fact-checking layer: does the essay get the physics right? The counterfactual pairs are multiple-choice questions with controlled variables (if I double the drop height, what happens to impact energy?). The single-video pattern tests are consistency checks (if you said the ball is rubber in paragraph 1, don’t call it metal in paragraph 3). The model passes only if it demonstrates understanding of cause-and-effect, not just pattern matching.

Key Concepts

• Counterfactual pairs: Two videos identical except for one physical variable. Example: same ball, same surface, but dropped from 1m vs 2m. The generated audio should differ in predictable ways (2m drop has ~1.4x the impact velocity, so louder and sharper sound). This isolates whether the model understands that specific physical relationship, removing confounds from other variables. It’s the scientific method applied to generative models—change one thing, measure the effect.

• Caption vs pixel reliance: V2A models take both video frames and optional text captions as input. The paper finds models lean heavily on captions to infer physics and semantics (what object, what material), but captions paradoxically hurt temporal alignment (when the sound should occur). This reveals a shortcut: models use text as a semantic lookup table rather than learning physics from visual dynamics. It’s like a student who reads the chapter summary instead of watching the experiment—they know *what should happen but not when or why.

• Physical correctness vs perceptual realism: An audio clip can sound plausible (high FAD score) while being physically wrong. Example: a small marble dropped from 1m might generate audio that sounds like a large bowling ball—perceptually realistic (it’s a plausible impact sound) but physically incorrect (wrong pitch, wrong energy). FlatSounds separates these dimensions: does the audio match human expectations (perceptual) *and match the actual physics (correctness)?

Framework Shift

Before (mainstream approach):        After (this paper):

Video --> V2A Model --> Audio        Video (controlled) --> V2A Model --> Audio
                |                         |                      |            |
                v                         v                      v            v
         Perceptual Metrics          Physical Ground Truth   Measure Physics
         (FAD, CLAP, MOS)            (size, height, material) (pitch, timing,
         "Does it sound good?"        "What should happen?"    energy, spectra)
                                                               "Did it happen?"

Evaluation = Human preference        Evaluation = Causal reasoning test

From judging outputs to auditing reasoning, the core shift is treating V2A as a physics simulator, not an audio synthesizer.

Expert Assessment

Problem choice: Real gap. The field has been chasing perceptual quality without checking physical grounding—this is the natural next question once models sound convincing. The timing is right: V2A models are good enough that we need to ask *what they’ve learned, not just how well they perform. The problem sits at the intersection of generative modeling and physical reasoning, which is increasingly important as we move toward embodied AI.

Method maturity: Clever and principled. The counterfactual design is textbook causal inference applied to generative models. The single-video pattern tests are a nice addition—they catch internal inconsistencies that counterfactuals miss. The metrics are domain-appropriate (pitch for size, onset detection for timing). One weakness: the benchmark is limited to impact sounds (drops, collisions). Extending to continuous sounds (pouring water, scraping) or complex scenes (multiple objects) would strengthen generalizability. The reliance on synthetic videos is both a strength (perfect control) and a limitation (domain gap to real videos).

Experimental integrity: Baselines are fair—they test multiple state-of-the-art models (Diff-Foley, FoleyCrafter, V2A-Mapper) under identical conditions. The ablation on captions is the paper’s strongest contribution: showing that captions help physics but hurt timing is a non-obvious finding that changes how we should train these models. One concern: the paper doesn’t report inter-annotator agreement for human preference tests, so we can’t assess how reliable those judgments are. The correlation between physics metrics and human preference (claimed in abstract) is mentioned but not quantified with numbers.

Writing quality: The paper is well-structured and the motivation is clear. The related work section could be tighter—it lists many V2A models but doesn’t clearly position what makes each one different. The results section buries the lead: the caption trade-off (physics vs timing) should be front and center, but it’s scattered across subsections. Figure 3 (the main results table) is dense and hard to parse—splitting it into separate tables for physical vs temporal metrics would improve readability. The discussion section is too short—it raises the “learn from pixels” challenge but doesn’t propose concrete paths forward.

Verdict: weak accept — Addresses a real gap with a principled method, but limited scope (impact sounds only) and missing details (human eval reliability, metric-preference correlation numbers) prevent it from being a strong contribution. The caption finding alone is worth publishing, but the paper needs tighter writing and broader evaluation to be a landmark benchmark.

Takeaways

For practitioners building V2A models: Don’t rely on captions as a crutch for physical reasoning. The paper shows captions improve semantic accuracy but degrade temporal alignment—this suggests a two-stage approach: use captions for *what (object identity, material) but learn when and how (timing, dynamics) from pixels. Consider disentangling semantic and temporal pathways in your architecture.

For benchmark designers: Counterfactual pairs are a powerful tool for auditing causal reasoning in generative models. The pattern transfers: if you’re evaluating any model that should respect physical laws (video prediction, robotics, simulation), construct controlled variations and measure whether outputs change in the expected direction. The single-video consistency tests are also generalizable—check whether a model maintains internal coherence across time or viewpoints.

For researchers in multimodal learning: The caption vs pixel trade-off is a microcosm of a broader issue: models take shortcuts through the easiest modality. Text is a compressed, high-level signal; pixels are raw and noisy. If both are available, models will lean on text even when pixels contain the ground truth. This suggests we need training objectives that explicitly force models to ground predictions in visual dynamics, not just semantic labels. Contrastive learning between caption-based and pixel-based predictions might help.

Specific technique to steal: The “directional trend” metric (does loudness increase as object approaches?) is simple but effective. You can apply this to any generative model that should respect monotonic relationships: does image brightness increase with light intensity? Does motion blur increase with velocity? Does audio pitch decrease with object size? Define the expected trend, measure whether the model follows it, and you have a physics-grounded evaluation metric.