Sequential Inference for Gaussian Processes: Bridging Signal Processing and Modern ML

Paper: 2604.28163 Authors: Daniel Waxman, Fernando Llorente, Petar M. Djurić Categories: eess.SP, cs.LG, stat.CO, stat.ML

The Gap

Signal processing has spent a century building tools for sequential data—filters, state-space models, adaptive algorithms. Machine learning has spent two decades building flexible models like Gaussian processes that excel at capturing complex patterns. But these two worlds speak different languages and make different assumptions. ML typically assumes batch data (all observations available at once), while SP lives in a streaming world where data arrives one point at a time and decisions must be made immediately.

Gaussian processes are powerful but computationally expensive: standard GP inference scales O(n³) in the number of observations. For streaming applications—sensor networks, real-time forecasting, adaptive control—this is a dealbreaker. Recent ML advances in sequential GP inference exist but are scattered across different communities, using inconsistent notation and lacking the systems perspective that SP practitioners need.

Classical SP tools          Modern ML models (GPs)
(streaming-native)          (batch-oriented)
        |                           |
        |                           |
        +---------------------------+
                    |
                    v
            [ THE GAP ]
        No unified framework
        for sequential GP inference
        from SP perspective
                    |
                    v
            This paper's bridge:
        Survey + organize recent
        sequential GP methods using
        SP concepts and notation
                    |
                    v
        Practitioners can now deploy
        GPs in real-time SP systems

The Increment

One sentence: Before this paper, sequential GP methods were fragmented across ML subfields; after, SP practitioners have a unified roadmap for deploying GPs in streaming systems.

Core Mechanism

This is a survey paper, not a methods paper—it doesn’t introduce new algorithms but organizes existing ones. The authors collect recent advances in sequential GP inference and reframe them using signal processing concepts: state-space representations, recursive updates, online learning, and computational efficiency.

The paper’s structure mirrors how an SP practitioner would approach the problem. Start with batch GP fundamentals (kernel functions, posterior inference, hyperparameter learning). Then introduce sequential scenarios: streaming data, concept drift, computational constraints. For each scenario, present the relevant algorithmic families—sparse approximations, inducing points, variational methods, particle filters—and show how they connect to classical SP tools like Kalman filters and recursive least squares.

Input: Scattered GP methods     Paper's organization:
from ML literature              
                                [Batch GP Foundations]
Method A (ML conf)                      |
Method B (stats journal)                v
Method C (robotics)            [Sequential Scenarios]
Method D (optimization)         /      |      \
        |                      /       |       \
        v                     v        v        v
   [Survey]              Streaming  Drift  Constraints
        |                     |       |        |
        v                     v       v        v
[Organized by            [Algorithmic Families]
 SP concepts]             - Sparse methods
        |                 - Inducing points
        v                 - Variational inference
Output: Unified           - Particle filters
framework with                 |
SP perspective                 v
                        [Connections to SP]
                        Kalman filters, RLS, etc.

Think of this paper as a Rosetta Stone for two research communities. On one side, you have hieroglyphics—ML papers using variational inference, ELBO objectives, and stochastic optimization. On the other side, you have Greek—SP papers using state-space models, recursive estimators, and adaptive filters. The Rosetta Stone doesn’t invent a new language; it shows that certain ML hieroglyphics mean the same thing as certain SP Greek phrases. A “sparse variational GP with inducing points” is structurally similar to a “reduced-rank Kalman filter.” An “online hyperparameter update” is doing the same job as “adaptive filter coefficient tracking.”

The value isn’t in the translation itself—it’s that once you see the correspondence, you can borrow tools from either side. An SP practitioner can now read a modern ML paper on streaming GPs and recognize the familiar structure underneath the unfamiliar notation. An ML researcher can see how their sequential inference method connects to decades of SP theory on stability, convergence, and computational complexity.

Key Concepts

• Sequential vs Batch Inference: In batch inference, you have all n data points upfront and compute the posterior in one shot—this costs O(n³) for matrix inversion. In sequential inference, data arrives one point at a time (or in small chunks), and you update your posterior incrementally. The goal is to avoid recomputing everything from scratch. Think of it like updating your mental model of traffic patterns: batch is watching a week of traffic footage and then forming an opinion; sequential is adjusting your model every time you drive, incorporating today’s observations into yesterday’s beliefs. Sequential methods trade exact inference for computational feasibility—you might use approximations (sparse representations, low-rank updates) to keep the cost manageable as data accumulates.

• Inducing Points: Standard GP inference requires storing and inverting an n×n covariance matrix over all observations. Inducing points are a smaller set of m “summary locations” (m << n) that approximate the full GP. Instead of conditioning on all n data points, you condition on these m inducing points, reducing computation from O(n³) to O(nm²). Picture a weather forecasting system: instead of storing temperature readings from every street corner in a city (n = millions), you place m = 100 weather stations at strategic locations. New observations are compared against these stations, not against every historical reading. The art is choosing where to place the stations (inducing point locations) and how to update them as data streams in. Poor placement loses information; good placement captures the essential structure with far less memory and compute.

• State-Space Representation of GPs: Many GP kernels (Matérn, exponential, etc.) can be rewritten as solutions to stochastic differential equations. This lets you represent a GP as a state-space model: a hidden Markov process with linear dynamics and Gaussian noise. Why does this matter? Because state-space models have 50+ years of SP theory behind them—Kalman filters, particle filters, smoothers. Once your GP is in state-space form, you can use these classical tools for sequential inference. The computational cost drops from O(n³) to O(1) per time step (constant memory, constant update time). The catch: only certain kernels admit exact state-space representations. For others, you need approximations. But when it works, it’s the gold standard for real-time GP inference.

Framework Shift

Before (mainstream ML approach):        After (this paper's SP lens):

All data available upfront              Data arrives sequentially
        |                                       |
        v                                       v
  [Batch GP Inference]                  [Recursive Updates]
   - Invert n x n matrix                 - O(1) per time step
   - O(n^3) cost                         - Constant memory
   - Exact posterior                     - Approximate posterior
        |                                       |
        v                                       v
  Works for small n,                    Works for streaming data,
  offline analysis                      real-time systems
                                                |
                                                v
                                        [Connection to SP Tools]
                                        Kalman filters, RLS,
                                        adaptive algorithms

From “compute everything at once” to “update incrementally,” the core shift is treating GPs as dynamical systems rather than static regression models.

Expert Assessment

Problem choice: This is a real gap. The ML and SP communities have been solving overlapping problems with different vocabularies for years. Sequential GP inference is mature enough in ML that a synthesis is overdue, and SP practitioners genuinely need this bridge to adopt modern methods. The timing is right—GPs are no longer exotic, and streaming data is everywhere.

Method maturity: As a survey, there’s no new method to evaluate. The contribution is organizational and pedagogical. The authors do a solid job connecting dots, but the paper leans heavily on existing work. The value is in the curation and the SP framing, not in algorithmic novelty.

Experimental integrity: No experiments—this is a tutorial-style survey. The paper includes illustrative examples but doesn’t claim empirical contributions. For what it is, that’s appropriate. A weakness: the paper could have included a more systematic comparison of computational costs and approximation quality across methods, even if borrowed from prior work.

Writing quality: The paper is well-structured and accessible, but it tries to cover too much ground. The sections on variational inference and particle filters feel rushed—more detail on when to use which method would help practitioners. The connections to SP are sometimes stated rather than demonstrated; a few worked examples showing the same problem solved with both ML and SP tools would make the bridge more concrete.

Verdict: weak accept — Useful synthesis for practitioners, but doesn’t push the research frontier. It’s a good tutorial, not a landmark paper.

Takeaways

• Steal the state-space trick: If your GP kernel is Matérn or exponential, rewrite it as a linear SDE and use Kalman filtering for O(1) sequential updates. This is production-ready and well-understood.
• Inducing points are your friend: For non-Markovian kernels or when state-space doesn’t apply, sparse GP methods with inducing points are the practical workhorse. The key is adaptive placement—update inducing point locations as data distribution shifts.
• Borrow SP stability analysis: ML papers often ignore convergence and stability in sequential settings. SP has decades of theory on when recursive estimators diverge. If you’re deploying sequential GPs, read the SP literature on adaptive filter stability—it applies directly.
• The real bottleneck is hyperparameters: Sequential inference for GP mean and variance is solved. The hard part is updating kernel hyperparameters online without retraining from scratch. The paper surveys methods but admits this is still an open problem. If you’re building a system, plan for periodic batch retraining or use priors that reduce hyperparameter sensitivity.