From Circuits to Dynamics: Understanding and Stabilizing Failure in 3D Diffusion Transformers

Paper: 2602.11130 Authors: Maximilian Plattner, Fabian Paischer, Johannes Brandstetter, Arturs Berzins Categories: cs.LG, cs.CV

Abstract

This work investigates a critical failure mode in 3D diffusion transformers used for surface completion from sparse point clouds. The authors discover that state-of-the-art models exhibit a phenomenon they term “Meltdown” - where arbitrarily small perturbations to input point clouds cause the generated output to fragment into disconnected pieces. Through mechanistic interpretability techniques, specifically activation-patching, they trace this failure to a single cross-attention activation in early denoising steps. The spectral entropy of this activation’s singular-value spectrum serves as a predictive signal for fragmentation. Building on this understanding, they develop PowerRemap, a test-time intervention that stabilizes the generation process across multiple architectures and datasets, achieving stabilization rates up to 98.3%.

Key Contributions

• Discovery and characterization of “Meltdown” - a catastrophic failure mode where minimal input perturbations cause output fragmentation in 3D diffusion transformers
• Localization of the failure mechanism to a specific early-stage cross-attention activation using activation-patching from mechanistic interpretability
• Identification of spectral entropy as a scalar proxy that predicts fragmentation events and tracks symmetry-breaking bifurcations in the reverse diffusion process
• Introduction of PowerRemap, a test-time control mechanism that stabilizes sparse point-cloud conditioning without retraining
• Demonstration that Meltdown is a systematic issue across state-of-the-art architectures (WaLa, Make-a-Shape), datasets (GSO, SimJEB), and sampling strategies (DDPM, DDIM)

Mechanistic Analysis: From Circuits to Bifurcations

The paper bridges two perspectives on diffusion model behavior: circuit-level analysis and dynamical systems theory. Using activation-patching, the authors systematically intervene on individual components to identify which activations are causally responsible for Meltdown. This reveals that a single cross-attention layer in early denoising steps acts as a critical control point.

The singular-value decomposition (SVD) of this attention activation provides interpretable structure. When the model is on track for successful generation, the singular values follow a characteristic distribution. However, when Meltdown occurs, the spectral entropy $H = -\sum_i \lambda_i \log \lambda_i$ (where $\lambda_i$ are normalized singular values) increases significantly, indicating a more uniform distribution of information across singular vectors.

From a dynamical systems perspective, this corresponds to a symmetry-breaking bifurcation in the reverse diffusion trajectory. The diffusion process can be viewed as a stochastic differential equation, and Meltdown represents a point where the trajectory diverges from the manifold of valid surface completions. The spectral entropy serves as an early warning signal for this bifurcation, rising before visible fragmentation occurs.

PowerRemap: Stabilizing Through Spectral Control

PowerRemap operates by modifying the singular-value spectrum of the critical cross-attention activation. The intervention applies a power transformation to the singular values:

$\tilde{\sigma}_i = \sigma_i^{\alpha}$

where $\alpha > 1$ amplifies the dominant modes and suppresses noise in the tail of the spectrum. This effectively “sharpens” the attention pattern, preventing the diffusion process from exploring unstable regions of the latent space.

The method is applied only at test time and requires no model retraining. Across experiments on WaLa and Make-a-Shape architectures with GSO and SimJEB datasets, PowerRemap achieves stabilization rates between 87.4% and 98.3%, dramatically reducing fragmentation events. Importantly, the intervention preserves generation quality on non-pathological inputs, showing that it specifically targets the failure mode without degrading normal operation.

Implications for Diffusion Model Robustness

This work reveals that even state-of-the-art diffusion models can harbor brittle failure modes that are triggered by imperceptible input changes. The universality of Meltdown across architectures and datasets suggests it may be an inherent challenge in conditioning diffusion models on sparse geometric data.

The success of mechanistic interpretability techniques in diagnosing and addressing this issue demonstrates the value of circuit-level analysis for understanding neural network behavior. By identifying specific activations responsible for failures, researchers can develop targeted interventions rather than architectural overhauls.

The connection between attention spectral properties and diffusion dynamics opens new avenues for monitoring and controlling generative processes. Spectral entropy and related metrics could serve as real-time diagnostic tools, enabling adaptive sampling strategies that detect and correct trajectory deviations before they manifest as visible artifacts.

Takeaways

1. State-of-the-art 3D diffusion transformers exhibit a systematic failure mode (“Meltdown”) where small input perturbations cause output fragmentation across multiple architectures and datasets.

2. Activation-patching from mechanistic interpretability successfully localizes the failure to a single early-stage cross-attention activation, demonstrating the power of circuit-level analysis for understanding model behavior.

3. The spectral entropy of attention activations serves as a predictive signal for generation failures, linking circuit-level mechanisms to dynamical systems accounts of diffusion trajectories.

4. PowerRemap provides an effective test-time intervention that stabilizes generation with up to 98.3% success rate without requiring model retraining, showing that targeted spectral control can address systematic failure modes.

5. The work establishes a methodology for understanding and correcting diffusion model failures through mechanistic analysis, potentially applicable to other generative modeling domains beyond 3D geometry.