How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech

Paper: 2606.20532 Authors: Nityanand Mathur, Hamees Sayed, Wasim Madha, Apoorv Singh, Sameer Khurana, Akshat Mandloi, Sudarshan Kamath Categories: cs.AI

The Gap

Existing style-captioned TTS systems (like VALL-E, CapSpeech) can generate speech conditioned on natural language descriptions, but they are treated as black boxes. You feed in “slow, breathy, sad” and get audio — but you have no idea which word caused which acoustic effect. Did “slow” lower F0? Did “breathy” increase energy in high frequencies? Prior work focused on controlling style via learned embeddings or explicit prosody tags, but never analyzed how the language itself steers the model’s internal attention. This paper is the first to open that black box for speech diffusion models.

[Problem: unknown per-word influence on acoustics] 
    |
    v
[Assumption: cross-attention scores encode word-level control]
    |
    v
[Method: DAAM attribution adapted to speech diffusion]
    |
    v
[Evidence: 3600 (caption, text) pairs; style tokens show low temporal variance]
    |
    v
[Conclusion: style conditioning peaks early in ODE and in deep layers]

The Increment

One sentence: Before this paper, we had no tool to trace how a style caption’s individual words affect the acoustic output of a speech diffusion model; after, we have a layer-by-layer, step-by-step heatmap that shows exactly which token controls which part of the waveform.

Core Mechanism

The method extends the Diffusion Attentive Attribution Maps (DAAM) framework from image generation to speech. It works on CapSpeech-TTS, a diffusion-based TTS model that takes two inputs: a text transcript (what to say) and a style caption (how to say it). The model uses cross-attention layers (25 of them) to let the style tokens influence the generation process over 24 ODE sampling steps.

For each combination of layer and ODE step, the method extracts the cross-attention weights between every style/text token and every patched mel-spectrogram frame. These weights form a per-token heatmap over time-frequency bins. The heatmaps are averaged across layers and steps to produce a final attribution map, and also analyzed per-layer and per-step to understand temporal dynamics. The authors then compute metrics like temporal variance (how much a token’s attention changes across time) and attention entropy (how evenly attention is spread) to characterize the behavior of different token types.

[Style Caption Tokens] ---> [Cross-Attn Layers (25)] ---> [Step-wise Attn Matrices]
[Text Transcript Tokens]-->  [ODE Steps (24)] --------->  [Per-Token Heatmaps]
                               v
                          [Aggregation: mean over layers/steps]
                               v
                          [Attribution Map: token vs time-freq bin]
                               v
                          [Metrics: temporal variance, entropy, F0 corr.]

Think of this as a film director’s call sheet. The style caption is the director’s note (“moody, slow, close-up”), and each word is an instruction to a different department. The diffusion model is the crew and camera team. The cross-attention layers are the department heads — each layer handles a different aspect of the aesthetic (early layers: global mood; deep layers: fine grain). The 24 ODE steps are the takes; in each take, the director (style tokens) adjusts the lighting and camera angles (acoustic features). The per-token heatmaps are like a post-production analysis showing that the word “moody” made the DP (F0) push for lower bass in the first three takes (early ODE steps), while “close-up” (breathiness) got the sound engineer to boost high-frequency hiss only in certain frames. Without this method, you’d only have the final movie — with it, you can read the director’s intent frame by frame.

Key Concepts

• Cross-attention attribution (DAAM for speech): Borrowed from text-to-image models, this technique takes the raw attention weights between condition tokens and output features, then normalizes and aggregates them to produce a saliency map. In speech, the output features are patches of the mel-spectrogram; each patch corresponds to a specific time and frequency bin. The attribution shows, for example, that the token “fast” has high attention at early time steps (fast speech rate) and low at later ones — a direct link between word and temporal region.

• Global vs. local conditioning: Style tokens (adjectives like “calm”, “rough”) show low temporal variance — their attention stays relatively constant across the whole utterance. Text tokens (content words like “hello”, “world”) show high temporal variance — they only influence the acoustic frames where they are spoken. This confirms that style acts as a global effect, while content is local. Prior work assumed this but never had direct evidence.

• Attention entropy: A measure of how concentrated a token’s attention is. Low entropy means only a few time-freq bins are influenced. The paper found that at layer 17, style tokens reach their peak importance (measured by attention weight magnitude), and simultaneously attention entropy hits its minimum. This means the network has the most selective, focused control exactly at the layer where style matters most — a lovely structural coincidence.

Framework Shift

Before (mainstream approach):                    After (this paper):

[Input: text + style caption]                   [Input: text + style caption]
       |                                                 |
       v                                                 v
[Black-box diffusion model]                      [Diffusion model + 25 x 24 attn extraction]
       |                                                 |
       v                                                 v
[Generated waveform]                             [Generated waveform] + [Per-token heatmaps]
                                                         |
                                                         v
                                                 [Analysis: temporal variance, entropy, F0/energy corr.]

From blind generation to interpretable generation — the core shift is moving from “what output” to “which input token caused which acoustic effect, and at what depth and time.”

Expert Assessment

Problem choice: Real gap. Style-conditioned TTS is a hot area (buzz around natural-language prosody control), and lack of interpretability is the main obstacle to debugging and user trust. The paper sits at the intersection of interpretability and controllable generation — a sensible trajectory.

Method maturity: Clever adaptation rather than brute force. DAAM is well-tested in vision; porting it to speech diffusion requires careful handling of 3D attention maps (batch, token, time-freq). The authors could have used simpler gradient-based attribution (e.g., integrated gradients), but cross-attention is more direct and computationally cheaper. No cleaner alternative is obvious.

Experimental integrity: Baselines are limited — they compare style tokens vs. content tokens, which is self-baseline but not a competing attribution method (none exist). The 3600 combinations are substantial. However, I’d question whether the observed patterns (low temporal variance) could be an artifact of token embedding distances rather than attentional structure. The authors did not ablate different prompt lengths or random control. Red flag: no statistical test for differences between token types (e.g., p-values).

Writing quality: The paper is dense but loses clarity in the method description — section 3 mixes notation across DAAM and speech domains without a clean translation table. If they rewrote section 3 with a single running example (e.g., track “warm” across layers and steps), the paper would be dramatically more accessible.

Verdict: weak accept — first-of-its-kind attribution for speech conditioning, but the empirical validation is thin on control experiments and statistical rigor.

Takeaways

• Three concrete steals: (1) The temporal variance metric can be reused in any conditional diffusion model to measure whether a condition token has global or local influence. (2) The attention entropy minimum at a specific layer is a diagnostic tool for over-specialization — if entropy dips too early, the model may ignore later refinements. (3) The per-token heatmap extraction pipeline (attention weights -> mel-spectrogram patches -> time-freq map) is directly transferable to other acoustic generative models like music or sound-effect synthesis with language prompts.