Structural Causal Bottleneck Models: Dimension Reduction That Respects Causality

Paper: 2603.08682 Authors: Simon Bing, Jonas Wahl, Jakob Runge Categories: stat.ML, cs.LG

The Gap

Causal inference in high-dimensional settings hits a wall: you need massive samples to estimate effects between, say, 10,000-dimensional images. Existing approaches split into two camps. Causal representation learning tries to discover latent variables that explain the data, but requires strong assumptions about the data-generating process. Causal abstraction learning maps high-level variables to low-level ones, but focuses on consistency rather than estimation efficiency. Neither directly addresses the practical question: what if causal effects only depend on a small part of the cause?

Problem: High-dim X -> High-dim Y needs huge samples
         |
         v
Assumption: Effect only depends on low-dim summary of X
         |
         v
Method: Learn bottleneck B = f(X), then Y = g(B, noise)
         |
         v
Evidence: Identifiable under conditions + works in transfer learning
         |
         v
Conclusion: Practical causal inference in high-dim via bottlenecks

The Increment

One sentence: Before, you either avoided high-dimensional causal inference or made strong assumptions about latent structure; now, you can explicitly model that effects flow through low-dimensional summaries while keeping estimation tractable.

Core Mechanism

The method introduces a bottleneck layer between cause and effect. Instead of modeling Y = f(X, noise) directly where X is high-dimensional, you split it: first compress X into a low-dimensional bottleneck B = h(X), then model Y = g(B, noise). The key constraint is that B must be sufficient for predicting Y—all information X has about Y flows through B.

The framework allows task-specific bottlenecks. If you care about predicting Y₁, you learn bottleneck B₁. If you care about Y₂, you learn B₂. Different effects can have different bottlenecks, each capturing only what matters for that particular causal relationship. This contrasts with representation learning, which seeks one universal latent space.

Estimation uses standard algorithms: train neural networks to learn the bottleneck function h and effect function g jointly, with a dimensionality constraint on B. The authors connect this to information bottleneck theory—you’re finding the minimal sufficient statistic of X for predicting Y. For identifiability, they show that under certain conditions (like having multiple contexts or interventions), you can uniquely recover the bottleneck structure.

High-dim cause X                    Effect Y
    |                                  ^
    |                                  |
    v                                  |
[Encoder h]                            |
    |                                  |
    v                                  |
Low-dim bottleneck B ----------------[Decoder g]
    |
    +---> (Sufficient for Y)
    +---> (Minimal dimension)

Think of it like airport security screening. Thousands of passengers (high-dimensional X) pass through, but security only checks a few key things: ID, boarding pass, metal detector reading (low-dimensional bottleneck B). The decision to let you board (effect Y) depends only on these summaries, not on your entire life history. Different checkpoints might check different things—international flights add passport checks, while domestic flights don’t. Each checkpoint has its own bottleneck tailored to its specific decision. The system works because these summaries are sufficient: they capture everything relevant about you for that particular decision, even though they throw away most information.

Key Concepts

• Causal Sufficiency vs Statistical Sufficiency: In statistics, a sufficient statistic captures all information in data about a parameter. Here, causal sufficiency means the bottleneck B captures all information X has about Y through the causal mechanism. Crucially, B might not be sufficient for predicting X from Y (the reverse direction), and it’s defined relative to a specific effect. Example: To predict if someone will get lung cancer (Y) from their medical history (X), smoking status and genetic markers (B) might be causally sufficient—you don’t need to know their childhood pet’s name, even though that’s in X.

• Task-Specific Bottlenecks: Unlike representation learning which seeks one “true” latent space, SCBMs allow different bottlenecks for different causal questions. If X causes both Y₁ and Y₂, you might have B₁ = h₁(X) for predicting Y₁ and B₂ = h₂(X) for predicting Y₂, where B₁ and B₂ are different. Example: From a patient’s full medical record (X), predicting diabetes risk (Y₁) might only need BMI and blood sugar (B₁), while predicting heart disease risk (Y₂) needs blood pressure and cholesterol (B₂). Same cause, different bottlenecks.

Framework Shift

Before (causal representation learning):     After (SCBMs):

X (observed) -----> Y (observed)            X (observed) -----> Y (observed)
    ^                   ^                       |                   ^
    |                   |                       v                   |
    |                   |                   [Bottleneck B]          |
    +---[Latent Z]-----+                       |                   |
                                               +-------------------+
Goal: Find universal Z                     Goal: Find minimal B per task
Assumption: Z generates X and Y            Assumption: Effect flows through B

[One sentence: From seeking universal latent representations that generate all variables, to finding minimal task-specific summaries through which causal effects flow.]

Expert Assessment

Problem choice: Real gap. High-dimensional causal inference is genuinely hard, and the bottleneck assumption is reasonable for many domains—think image classification where only certain features matter, or genomics where effects operate through pathways. The positioning against causal representation learning is smart: they’re solving a different problem (estimation efficiency) rather than competing directly.

Method maturity: Mostly repackaging existing ideas (information bottleneck, sufficient statistics) in a causal framework, but that’s valuable. The identifiability analysis is solid though not groundbreaking—it requires multiple contexts or interventions, which is standard. The connection to information bottleneck is natural but not deeply exploited; they could push harder on rate-distortion tradeoffs.

Experimental integrity: Experiments are illustrative rather than comprehensive. The transfer learning demo is compelling but limited to synthetic data and one real dataset. Missing: comparisons with recent causal representation learning methods, ablations on bottleneck dimension selection, and failure cases. The paper reads more like a framework proposal than an empirical validation.

Writing quality: Clear exposition of the framework, but the related work section undersells how much this overlaps with existing sufficient dimension reduction and information bottleneck literature. The identifiability section is dense—moving some proofs to appendix and adding intuitive examples would help. The experimental section feels rushed; spending more space on when bottlenecks help vs hurt would strengthen the paper.

Verdict: weak accept — Solid conceptual contribution that clarifies the role of bottlenecks in causal inference, but needs stronger empirical validation and clearer positioning relative to existing dimension reduction methods.

Takeaways

Steal the task-specific bottleneck idea: when building causal models, don’t assume you need one representation for everything. Learn different low-dimensional summaries for different downstream tasks. This is especially useful in transfer learning—train a bottleneck on source domain, freeze it, adapt only the effect model on target domain with few samples.

Practical trick: when estimating high-dimensional causal effects, add an explicit bottleneck layer with dimensionality constraint rather than hoping your neural network learns sparse representations implicitly. This gives you control and interpretability.

The identifiability conditions tell you when you can trust your bottleneck: you need variation across contexts (different distributions) or interventions. If you only have observational data from one distribution, your bottleneck might be arbitrary.