SVDA: Making Vision Transformers Interpretable for Monocular Depth Estimation

Paper: 2602.11005 Authors: Vasileios Arampatzakis, George Pavlidis, Nikolaos Mitianoudis, Nikos Papamarkos Categories: cs.CV

Abstract

This paper tackles the black-box nature of self-attention in Vision Transformers for monocular depth estimation. The authors introduce SVD-Inspired Attention (SVDA), a spectrally structured attention mechanism that embeds a learnable diagonal matrix into query-key interactions. Unlike standard attention, SVDA produces intrinsically interpretable attention maps and enables quantitative analysis through six spectral indicators: entropy, rank, sparsity, alignment, selectivity, and robustness. Experiments on KITTI and NYU-v2 datasets demonstrate that SVDA maintains competitive accuracy while revealing consistent patterns in how attention evolves during training—insights previously hidden in conventional Transformer architectures.

Key Contributions

• Introduction of SVDA, the first spectrally structured attention formulation for dense prediction tasks that decouples directional alignment from spectral modulation
• Development of six quantifiable spectral indicators that measure attention behavior: entropy, rank, sparsity, alignment, selectivity, and robustness
• Demonstration that interpretability can be built into the architecture rather than approximated post-hoc
• Empirical validation on KITTI and NYU-v2 showing preserved or improved accuracy with minimal computational overhead
• Discovery of consistent cross-dataset and depth-wise patterns in attention organization during training

Methodology: Spectral Decomposition of Attention

The core innovation lies in reformulating the attention mechanism through a spectral lens. Standard self-attention computes:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

SVDA introduces a learnable diagonal matrix $\Sigma$ into the normalized query-key interaction:

$A_{\text{SVDA}} = \text{softmax}(\hat{Q}\Sigma\hat{K}^T)$

where $\hat{Q}$ and $\hat{K}$ are normalized queries and keys. This diagonal matrix acts as a spectral modulator, controlling the contribution of different singular value components. The formulation draws inspiration from Singular Value Decomposition (SVD), where $\Sigma$ represents the spectrum of the attention map.

By embedding this structure directly into the attention computation, SVDA enables the extraction of spectral properties without requiring post-hoc decomposition. The learnable nature of $\Sigma$ allows the network to adapt its spectral characteristics during training while maintaining interpretability.

The authors integrate SVDA into the Dense Prediction Transformer (DPT) architecture, replacing standard multi-head attention blocks. Each attention head learns its own diagonal matrix, allowing different heads to specialize in different spectral patterns.

Spectral Indicators: Quantifying Attention Behavior

SVDA unlocks six spectral indicators that provide quantitative insights into attention dynamics:

1. Entropy: Measures the uniformity of attention distribution. High entropy indicates diffuse attention across many tokens, while low entropy suggests focused attention on specific regions.

2. Rank: Approximates the effective dimensionality of the attention map. Lower rank indicates that attention can be represented with fewer components, suggesting structured patterns.

3. Sparsity: Quantifies the concentration of attention weights. Sparse attention focuses on a small subset of tokens, while dense attention distributes weights broadly.

4. Alignment: Measures the correlation between query and key representations. High alignment indicates strong directional correspondence between spatial locations.

5. Selectivity: Captures the discriminative power of attention—how distinctly the mechanism differentiates between relevant and irrelevant tokens.

6. Robustness: Assesses the stability of attention patterns across perturbations or variations in input.

These indicators are computed directly from the spectral properties of the attention maps during forward passes, requiring no additional training objectives. The authors observe that these metrics reveal consistent patterns across datasets and depth ranges, suggesting fundamental organizational principles in how Transformers process spatial information for depth estimation.

Experimental Results and Analysis

The authors evaluate SVDA on two standard benchmarks: KITTI (outdoor autonomous driving scenes) and NYU-v2 (indoor environments). The results demonstrate several key findings:

Performance: SVDA achieves comparable or slightly better accuracy than standard DPT across standard metrics (absolute relative error, RMSE, threshold accuracy). On KITTI, SVDA shows marginal improvements in $\delta_1$ accuracy (percentage of pixels with relative error<1.25), while on NYU-v2, performance remains statistically equivalent.

Computational Overhead: The addition of learnable diagonal matrices introduces minimal computational cost—approximately 2-3% increase in FLOPs and negligible impact on inference time. This makes SVDA practical for real-world deployment.

Cross-Dataset Consistency: The spectral indicators reveal remarkably consistent patterns across KITTI and NYU-v2 despite their different characteristics. Early layers exhibit high entropy and low rank, gradually transitioning to more selective, lower-entropy patterns in deeper layers. This suggests universal organizational principles in how Transformers build depth representations.

Depth-Wise Patterns: Analysis of attention behavior at different depth ranges shows that near-field regions (close objects) induce higher selectivity and lower entropy, while far-field regions exhibit more diffuse attention. This aligns with the intuition that nearby objects require precise localization while distant regions benefit from broader contextual aggregation.

Training Dynamics: Tracking spectral indicators during training reveals that attention organization stabilizes relatively early (within 20-30% of total epochs), with rank and sparsity converging before entropy and selectivity. This suggests a hierarchical learning process where structural patterns emerge before fine-grained discriminative features.

Implications for Interpretable AI

SVDA represents a paradigm shift in how we approach interpretability in deep learning. Rather than treating interpretability as a post-hoc analysis problem, the work demonstrates that interpretable mechanisms can be embedded directly into architectures without sacrificing performance.

The spectral indicators provide a quantitative vocabulary for discussing attention behavior, moving beyond qualitative visualizations. This enables systematic comparison across architectures, datasets, and tasks—a critical step toward understanding what makes certain attention patterns effective.

For monocular depth estimation specifically, the insights reveal that Transformers organize spatial information through a progression from broad contextual aggregation to focused local refinement. This understanding could inform architectural design choices, such as where to place attention blocks or how to initialize spectral parameters.

More broadly, the SVDA framework is applicable beyond depth estimation to any dense prediction task (semantic segmentation, optical flow, surface normal estimation). The spectral formulation is task-agnostic and could be integrated into various Transformer architectures.

The work also raises important questions: Can we design training objectives that explicitly optimize for desirable spectral properties? Could spectral indicators serve as regularizers to improve generalization? How do spectral patterns differ across vision tasks, and what does this reveal about task-specific inductive biases?

Takeaways

1. SVDA introduces the first spectrally structured attention mechanism for dense prediction, making Transformer attention intrinsically interpretable rather than requiring post-hoc analysis.

2. Six spectral indicators (entropy, rank, sparsity, alignment, selectivity, robustness) provide quantitative measures of attention behavior that reveal consistent patterns across datasets and depth ranges.

3. Interpretability does not require sacrificing performance—SVDA maintains or slightly improves accuracy on KITTI and NYU-v2 while adding only 2-3% computational overhead.

4. Attention organization follows universal patterns: early layers use high-entropy, low-rank aggregation while deeper layers transition to selective, focused patterns.

5. The spectral formulation is architecture-agnostic and applicable to any Transformer-based dense prediction task, opening avenues for interpretable computer vision models.

6. Training dynamics show that structural attention patterns (rank, sparsity) stabilize before discriminative features (selectivity), suggesting hierarchical learning processes.

7. By shifting attention from opaque mechanism to quantifiable descriptor, SVDA establishes a principled framework for understanding and designing transparent deep learning models.