The Importance of Phase in Neural Representations: An Internal Oppenheim-Lim Test of Image Classifiers

Paper: 2606.17037 Authors: Alper Yıldırım Categories: cs.CV, cs.AI, cs.LG

The Gap

Oppenheim and Lim (1981) showed that natural images are almost entirely identified by their Fourier phase; magnitude alone is nearly unrecognizable. This asymmetry is a known property of pixel-level structure, but nobody had asked whether it survives inside the hidden layers of trained classifiers. Prior work on feature importance used saliency maps, ablation, or linear probes – these are correlational, not causal. They tell you that a feature matters, but not which component (phase vs magnitude) carries the identity. The gap: no causal test for the phase/magnitude role within the network itself.

[Problem: Do hidden layers replicate image-level phase dominance?]
    |
    v
[Assumption: Phase/sign carries identity, magnitude is background]
    |
    v
[Method: Internal Oppenheim-Lim test - transplant phase of image A
         onto magnitude of image B at a chosen layer L]
    |
    v
[Evidence: In ViT, GFNet, PRISM2D -> prediction follows phase donor.
          In ResNet -> ReLU masks phase; need pre-ReLU intervention,
          then same result. Deleting image-specific magnitude barely drops accuracy.]
    |
    v
[Conclusion: Yes - all architectures share a phase/sign identity code.
            Differences arise from rectification and readout geometry,
            explaining the texture-shape gap between CNNs and attention models.]

The Increment

One sentence: Before this paper, we only knew that pixel-phase matters for images; now we know that the *same principle holds inside neural network representations, and the shape-vs-texture preference of different architectures is a side effect of how they expose the phase code.

Core Mechanism

The method is a causal intervention on hidden-layer activations. Pick any intermediate layer L. Given two input images, run the network up to L for each, obtaining two activation tensors. For each spatial location and channel, decompose into magnitude (absolute value) and phase (for complex representations like PRISM2D) or sign (for real-valued activations after ReLU). Now construct a hybrid tensor: take the phase/sign of image A and the magnitude of image B, combine them element-wise, and feed the result into the remaining layers (L+1 onward). Record which image the final prediction matches. If the prediction follows the phase donor, then phase/sign carries the identity at that layer. A stronger test: delete all image-specific magnitude by setting it to a constant (e.g., the layer-wide mean) and measure accuracy change.

[Input A] --- [Layer L extract] --- [Compute phase/sign(A)]
                                          |
                                          |   combine
                                          v
     [Input B] --- [Layer L extract] --- [Compute magnitude(B)] -> [Hybrid tensor]
                                                                        |
                                                                        v
                                                                  [Forward L+1..end]
                                                                        |
                                                                        v
                                                                  [Prediction -> follows A or B?]

Structural metaphor: Think of neural layer activations as a stage play. The phase/sign is the actor’s performance – the gestures, the tone, the timing. The magnitude is the scenery – the backdrop, the lighting levels. You can swap the scenery completely (set magnitude to a constant) and the audience still knows who’s on stage. But if you give the scenery from one play to a different actor (swap phase onto someone else’s magnitude), the audience will think they’re watching the actor’s play. In this paper, the “audience” is the network’s readout head, and the play is the classification. The surprising result: even in deep layers, the actor (phase/sign) alone tells the story; the scenery (magnitude) is near-irrelevant.

For ResNet, the ReLU activation acts like a mask that clips all negative “gestures” to zero, effectively hiding the actor’s expression. So you have to look before the mask (pre-ReLU), where the full sign code still exists. Once you do, ResNet behaves just like the others.

Key Concepts

• Phase / Sign decomposition: For complex-valued representations (e.g., Fourier features in GFNet or PRISM2D), the phase is the complex argument. For real-valued activations, replace phase with the sign (positive/negative). The idea: sign preserves the *direction of deviation from the mean, while magnitude (absolute value) captures only how far from zero. The paper shows that sign carries almost all identity information; magnitude is just a global scale.

• Causal intervention vs correlation: Earlier methods (saliency, probing) only show that a feature is *correlated with the output. By physically swapping components and seeing the prediction flip, this paper proves causation. If prediction follows phase, then phase is directly controlling the output – a much stronger statement.

• The texture-shape gap: Convolutional networks (like ResNet) are biased toward texture; vision transformers (like ViT) toward shape. This paper provides a mechanistic account: both architectures encode identity in phase/sign, but ReLU in CNNs discards the sign of negative activations, forcing the readout to rely on a channel-wise spatial average (a DC component) that is more sensitive to texture-like statistics. Transformers keep the full sign, so they can exploit shape-correlated phase patterns.

Framework Shift

Before (mainstream approach):                       After (this paper):
+-----------------------------+                    +-------------------------------+
| Train model, freeze, then   |                    | Train model, decompose hidden  |
| use saliency / probing to   |                    | layer activations into phase   |
| infer which features matter |                    | and magnitude, then swap or    |
| (correlational)             |                    | delete magnitude (causal).     |
+-----------------------------+                    +-------------------------------+
| Result: "Feature X is       |                    | Result: "Phase/sign causes     |
| important for class Y."     |                    | identity; magnitude is         |
| No insight into *what       |                    | disposable. The texture-shape  |
| component* of the feature   |                    | gap is a byproduct of how      |
| carries identity.           |                    | different architectures expose |
+-----------------------------+                    | the same phase code."          |
                                                   +-------------------------------+

One sentence: From correlational feature scoring to causal component swapping, the core shift is disentangling phase from magnitude as the true causal carrier of identity.

Expert Assessment

Problem choice: Excellent – the Oppenheim-Lim phenomenon is a classic result in signal processing, and extending it to neural representations is both elegant and overdue. The gap is real, not manufactured.

Method maturity: Clever and minimal. No heavy machinery – just a decomposition and a swap. The authors resist the urge to over-engineer. However, a simpler approach (e.g., just zeroing out magnitude) would have been even cleaner; they did that as a control, which is good.

Experimental integrity: Fair baselines (including a DC-only control that rules out trivial magnitude independence). The ResNet nuance is correctly handled. One red flag: the paper tests only four architectures (PRISM2D, GFNet, ViT-B/16, ResNet-50). A broader sweep (ConvNeXt, MLP-Mixer, Swin) would strengthen the claim of universality.

Writing quality: Solid, but the abstract and introduction are dense; they assume familiarity with Fourier analysis and causal testing. The section on ResNet’s ReLU masking is the best-written part – clear and convincing. If I could rewrite one section, it would be the results: more space for tables showing accuracy after deletion vs after swap, and a clearer visual of the hybrid tensor construction.

Verdict: weak accept – A refreshing causal perspective on a classic question. The core discovery (phase dominance inside networks) is novel and mechanistically informative, even if the experimental scope is narrower than ideal.

Takeaways

• Measurement technique: If you work on interpretability, adopt the phase/sign decomposition + swap as a standard tool for checking which component of a hidden representation carries semantics. It’s model-agnostic and cheap.
• Design insight: If you want a model to use shape (like ViT), avoid nonlinearities that discard sign (e.g., ReLU). Consider using pre-activation gating or complex-valued layers to preserve phase information.
• For practitioners in other domains (NLP, speech): The same decomposition applies to hidden states. You can test whether token-level phase (or sign) dominates task identity. This could reveal which architectures are optimal for tasks where phase patterns (e.g., word order, prosody) are critical.