Deep Gaussian Processes Meet DAGs: Compositional Uncertainty on Graph Structures

Paper: 2607.09645 Authors: Federico L. Perlino, Oliver Hamelijnck, Adam M. Johansen, Theodoros Damoulas Categories: stat.ML, cs.LG, math.ST, stat.CO, stat.ME

The Gap

So here’s the situation: a lot of real-world systems — causal mechanisms, multi-fidelity simulators, gene-regulatory networks — are naturally modeled as compositions of functions along a DAG. Think: transcription factor A activates gene B, which together influence protein C. Each of these functions gets observed partially, noisily, and at different sampling rates.

The problem is that standard Deep Gaussian Processes (DGPs) assume a chain structure: layer 1 feeds layer 2 feeds layer 3. That’s fine for feedforward neural networks, but it completely ignores the graph topology. Collider nodes — where two parent paths converge — require “explaining away” behavior (observing one parent changes beliefs about the other), which chain DGPs simply can’t represent. Meanwhile, multi-fidelity methods and causal inference tools each handle fragments of this problem, but nobody has put together a principled probabilistic framework that respects DAG structure, propagates compositional uncertainty, and handles heterogeneous observations.

Dunlop et al. (2018) noticed something interesting about how input connections affect deep GP behavior, but never proved it formally. The theoretical understanding of when and why information gets lost in deep compositions was incomplete.

Problem: Functions composed along DAGs
         observed partially & noisily
         |
         +--- Chain DGPs ignore graph topology
         |
         +--- No explaining-away for colliders
         |
         +--- Dunlop's observation unproven
              |
              v
Assumption: Place GP priors on each node function
            Respect DAG dependency structure
              |
              v
Method: DAG-DGP + structured variational approx
        that preserves graph dependencies
              |
              v
Evidence: Theory: almost-sure lower bounds on
          information preservation depth
          Experiments: 3 tasks, state-of-art
              |
              v
Conclusion: DAG topology matters for
            compositional uncertainty
            Graph-aware inference is tractable

The Increment

One sentence: Before this paper, deep GPs could only compose functions along chains; after, they can compose along arbitrary DAGs with principled uncertainty propagation and collider reasoning.

Core Mechanism

The setup is clean: you have a DAG where each node represents a latent function. Some nodes have observations (noisy, possibly heterogeneously sampled). The challenge is to learn all the latent functions jointly while respecting the graph structure and propagating uncertainty through compositions.

At each node, you place a GP prior. The inputs to a node’s GP come from its parents’ outputs. For a node with multiple parents, the input is a concatenation of parent outputs. This means the “width” of the input space grows with in-degree, and the posterior at each node must account for uncertainty flowing from all upstream sources.

The variational approximation is where the cleverness lives. Instead of a mean-field factorization that would break graph dependencies, they construct a structured variational family that retains the DAG topology. Inducing points are placed at each node, but their joint distribution respects the graph structure. For collider nodes, this naturally captures explaining-away: conditioning on observations downstream of a collider creates dependencies between the variational posteriors at its parent nodes.

Node A (GP)          Node B (GP)
    |                    |
    |    x_A             |    x_B
    v                    v
  f_A(x_A)          f_B(x_B)
    |                    |
    +--------+-----------+
             |
             v
       Node C (GP)  <-- collider: f_C(f_A, f_B)
             |
             v
       Observations y_C
             |
     (explaining away: y_C updates
      beliefs about both f_A and f_B)

Structural Metaphor

Think of a DAG-DGP like a water distribution network in a city.

Each node is a water treatment plant. The pipes (edges) carry water (function outputs) between plants. Some plants have wells (observations) feeding raw water in, but most only receive water from upstream plants.

A fan-out node is like a reservoir that splits water to multiple downstream neighborhoods. A collider node is like a treatment plant that receives water from two different sources and mixes them — if the mixed water tests dirty (observation), you need to figure out which upstream source caused the contamination. That’s explaining away.

Now, the variational approximation is like a network-wide monitoring system. You place sensors (inducing points) at each plant, but the monitoring system understands the network topology. If plant C’s output degrades, the system doesn’t just blame C locally — it traces back through the pipes to update its assessments of plants A and B. A naive monitoring system (mean-field) would treat each plant independently and miss this.

The prior collapse theory answers: how deep can your pipe network go before the water at the far end becomes indistinguishable regardless of what you put in at the well? They prove that for broad kernel classes, the network preserves input information at infinitely many depths — so your network doesn’t become a useless constant function as it deepens.

Key Concepts

• Prior collapse: As you stack GP layers, the outputs can converge to degenerate distributions that don’t depend on the inputs anymore. Imagine a game of telephone where after enough rounds, every message becomes the same mush. The paper proves that for reasonable kernel choices, this doesn’t happen at every depth — there are infinitely many depths where the GP still “sees” the input. Concrete example: with squared-exponential kernels, the composition f_n(f_{n-1}(…f_1(x)…)) still varies meaningfully with x for infinitely many n.

• Explaining away at colliders: In a DAG, a “collider” is a node with multiple parents whose paths converge there. In a Bayesian network, observing the collider creates dependence between its parents. Concretely: if disease C requires both genes A and B, and you observe C, then learning A is present makes you less confident B is absent (because C was observed). The variational method in this paper preserves this behavior, whereas factorized approximations would break it.

• Structured variational family: Instead of assuming all latent functions are independent in the posterior (mean-field), the variational distribution respects the DAG. The joint q(f_1, …, f_K) factors according to the graph: q(f_1) ** q(f_2|parents) * … This is more expensive but captures the dependencies that matter for correct uncertainty propagation.

Framework Shift

Before (mainstream approach):        After (this paper):

Layer 1 -> Layer 2 -> Layer 3       Node A ----+---> Node D
   (chain structure only)                  \   |
                                      Node B --+---> Node E
                                            |
                                      Node C -------> Node F
                                      (arbitrary DAG structure)

Variational: mean-field              Variational: graph-structured
q(f1) * q(f2) * q(f3)               q(fA) * q(fB|fA) * q(fC|...)
(breaks dependencies)                (preserves dependencies)

Colliders: not modeled               Colliders: explaining-away captured

From chain compositions to DAG compositions, the core shift is respecting graph topology in both the prior and the inference.

Expert Assessment

Problem choice: This is a real gap. The intersection of DAG structure, function composition, and Bayesian nonparametrics has been underexplored. Causal inference people use DAGs but don’t typically think about GP priors on mechanisms. GP people build deep models but assume chains. Multi-fidelity folks have similar problems but ad hoc solutions. The paper sits at a genuine crossroads.

Method maturity: The structured variational approximation is the right idea — mean-field would be a non-starter for colliders. However, the complexity scales with in-degree (concatenating parent outputs) and with the number of inducing points per node. For sparse DAGs this is fine, but dense DAGs could blow up. The paper doesn’t address scalability limitations head-on. Also, the inducing point framework is by now standard; the novelty is in how the graph structure is incorporated, which is clever but not earth-shattering.

Experimental integrity: Three tasks cover different regimes: a synthetic latent collider (verifying explaining-away), a protein signalling network (real biological DAG), and multi-fidelity heavy-ion collision emulation (applied physics). The baselines are reasonable — standard DGPs and GP-based multi-fidelity methods. State-of-the-art claims appear supported. One concern: the protein signalling network is fairly small, so it’s unclear how this scales to larger graphs. The heavy-ion collision task is the most compelling practical demonstration.

Writing quality: The theory section is dense but well-structured. The prior-collapse results are genuinely novel. However, the intuition-building between theorems is thin — a reader not steeped in GP theory will struggle. Section 3 (method) could benefit from a running example that develops alongside the formalism. The experiments section does its job but doesn’t linger on failure cases or limitations.

Verdict: weak accept — solid contribution at a genuine intersection, with meaningful theory and practical demonstrations, but limited scalability evidence and dense presentation hold it back from a strong accept.

Takeaways

Three things worth stealing:

1. Graph-aware variational families: If your problem has DAG structure, don’t default to mean-field. The structured variational approximation that factors according to the graph is a template you can apply to other models beyond GPs (e.g., normalizing flows on DAGs).

2. Collider reasoning as a benchmark: If you’re building compositional probabilistic models, test whether they can do explaining-away at colliders. It’s a simple diagnostic that catches broken independence assumptions.

3. Prior collapse as a design criterion: Before building a deep compositional model, check whether information survives through the composition. The kernel conditions identified here (roughly: kernels that don’t shrink outputs too aggressively) give concrete guidance for choosing activation/kernel functions in deep architectures.