Stop Predicting Pixels: Why Latent Space Learning Beats Next-Frame Prediction for Physical Systems

Paper: 2603.13227 Authors: Helen Qu, Rudy Morel, Michael McCabe, Alberto Bietti, François Lanusse, Shirley Ho, Yann LeCun Categories: cs.LG, cs.CV

The Gap

The field has been chasing next-frame prediction as the holy grail for learning from physical systems. Methods like PDE-based neural operators, physics-informed neural networks, and video prediction models all optimize for pixel-level accuracy in predicting future states. But this creates three problems: (1) training is computationally expensive, (2) errors compound during autoregressive rollout, and (3) pixel accuracy doesn’t guarantee physical understanding.

The real gap: scientists don’t just want pretty animations. They want to extract governing parameters (viscosity, diffusion coefficients), predict long-term statistics, and understand physical mechanisms. No one has systematically asked whether next-frame prediction is even the right objective for these downstream scientific tasks.

Problem: Next-frame prediction ≠ physical understanding
   |
   v
Assumption: Representations useful for science
            can be learned without pixel prediction
   |
   v
Method: Evaluate self-supervised methods on
        downstream scientific tasks (parameter estimation)
   |
   v
Evidence: Latent-space methods (JEPAs) outperform
          pixel-level methods on these tasks
   |
   v
Conclusion: Optimize for what you actually need,
            not for pixel reconstruction

The Increment

One sentence: Before this paper, we trained models to predict pixels and hoped they learned physics; after, we have evidence that optimizing latent representations directly is better for actual scientific inference.

Core Mechanism

The method is an evaluation framework, not a new architecture. They take existing self-supervised learning approaches and test them on a battery of scientific tasks. The key components: (1) a frozen encoder trained with various self-supervised objectives (MAE, SimCLR, JEPA variants), (2) simple linear probes or small MLPs trained on top for downstream tasks, (3) evaluation on parameter estimation (e.g., predicting Reynolds number from fluid flow), long-term statistics prediction, and other physics-grounded tasks.

Data flows like this: raw spatiotemporal data (fluid simulations, climate data) → encoder → frozen representations → task-specific probe → scientific prediction. The critical operation is the separation of representation learning from task-specific learning, which isolates what the encoder actually learned about physics.

They compare methods across two axes: (1) pixel-level vs latent-space objectives, and (2) generative vs discriminative approaches. The surprise: methods that never reconstruct pixels (JEPAs) often win.

Input: Spatiotemporal physical data
   |
   v
[Encoder] <-- Trained with self-supervised objective
   |          (MAE, SimCLR, JEPA, etc.)
   v
Frozen representations (latent vectors)
   |
   v
[Linear probe] <-- Trained on downstream task
   |              (parameter estimation, etc.)
   v
Scientific prediction (viscosity, Reynolds #, etc.)

Key insight: Freeze encoder, evaluate on science tasks

Think of this like testing different microscopes by seeing which one helps biologists identify cell types fastest, rather than which one produces the prettiest images. The encoder is the microscope, the downstream task is cell identification, and the frozen representations are what the microscope reveals. Some microscopes (pixel predictors) optimize for image quality but lose information useful for identification. Others (JEPAs) optimize for preserving distinguishing features in a compressed form, which turns out to be more useful for the actual scientific task. The metaphor maps: microscope → encoder, image quality → pixel accuracy, distinguishing features → latent representations, cell identification → parameter estimation.

Key Concepts

• Joint Embedding Predictive Architecture (JEPA): Instead of predicting raw pixels, predict the representation of future frames in latent space. Imagine you’re learning to forecast weather. A pixel predictor tries to draw every cloud pixel-by-pixel. A JEPA learns abstract features (pressure systems, temperature gradients) and predicts how those features evolve. When you later need to estimate a parameter like humidity, the abstract features are more useful than pixel patterns. JEPAs use two encoders (one for context, one for target) and predict target representations from context representations, avoiding the computational cost of pixel generation.

• Representation collapse: When a model learns to map all inputs to the same output, making the representation useless. Like a student who answers every exam question with “I don’t know” - technically a valid strategy to minimize effort, but learns nothing. Self-supervised methods fight this with tricks like contrastive losses (push different inputs apart) or variance regularization (ensure representations spread out). The paper shows that methods resistant to collapse (like JEPAs with variance-covariance regularization) learn better physics representations.

Framework Shift

Before (mainstream approach):        After (this paper):

Train model:                         Train model:
[Physical data]                      [Physical data]
      |                                    |
      v                                    v
[Next-frame predictor]               [Self-supervised encoder]
      |                                    |
      v                                    v
[Pixel-level loss]                   [Latent-space objective]

Evaluate:                            Evaluate:
[Rollout accuracy]                   [Freeze encoder]
[Pixel MSE over time]                      |
                                           v
                                     [Train probe on science task]
                                           |
                                           v
                                     [Parameter estimation accuracy]

Focus: Can you predict pixels?       Focus: Can you extract physics?

From “predict the future perfectly” to “learn representations that support scientific inference”, the core shift is evaluation on what scientists actually need rather than what’s easy to measure.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The field has indeed been myopically focused on next-frame prediction, and the authors correctly identify that this doesn’t align with scientific goals. The problem sits at a productive intersection - it’s not just about better architectures, but about rethinking objectives. However, the framing could be sharper: they sometimes conflate “physics-grounded” with “useful for downstream tasks”, which aren’t quite the same thing.

Method maturity: This is primarily an evaluation paper, not a methods paper, which is fine. The experimental design is solid - they compare multiple baselines fairly on the same tasks. However, there’s a missed opportunity: they don’t deeply investigate *why JEPAs work better. Is it the latent-space objective? The architecture? The training dynamics? More ablations would strengthen the claims. The choice to use simple probes is smart (isolates representation quality) but also limits insight into what the representations actually encode.

Experimental integrity: Baselines are reasonable, though I’d want to see more physics-specific methods (e.g., Hamiltonian neural networks, Lagrangian neural networks). The tasks are well-chosen and diverse. One concern: the paper doesn’t discuss computational costs much - if JEPAs are 10x cheaper to train, that’s a huge practical advantage that should be front and center. The results are convincing but not overwhelming - JEPAs win, but not by massive margins in all cases.

Writing quality: The abstract and introduction are clear, but the related work section is too long and doesn’t sufficiently distinguish this work from prior representation learning papers. The results section would benefit from more analysis of failure cases - when do pixel predictors win, and why? The discussion of what makes a representation “physics-grounded” is hand-wavy and could be formalized. If I were reviewing, I’d ask for a clearer definition of this term and metrics to measure it directly.

Verdict: weak accept - solid empirical work that challenges a field assumption, but lacks the theoretical depth or overwhelming empirical evidence for a strong accept. The reframing is valuable, but the execution could be sharper.

Takeaways

For practitioners working with spatiotemporal data (not just physics): consider whether your evaluation metric aligns with your actual goal. If you need to extract parameters or make long-term predictions, optimizing for next-frame pixel accuracy might be the wrong objective. Try latent-space methods like JEPAs, especially if you’re working with high-dimensional data where pixel reconstruction is expensive.

Specific technique to steal: the frozen encoder + simple probe evaluation protocol. This cleanly separates representation quality from task-specific learning and is much cheaper than full fine-tuning. Use it to compare different pretraining strategies on your domain.

For researchers: this paper opens a question more than it answers one. What properties of latent representations make them “physics-grounded”? Can we design objectives that explicitly optimize for these properties? The gap between pixel prediction and scientific utility is real, but we still don’t have a principled theory of what makes a good representation for physical reasoning.