JEPA-Anything: Learning Predictive Models across Different Worlds

Paper: 2609.20800 Authors: Taoyong Cui, Zhongyao Wang, Xinyue Xu, Weiyang Liu, Zhaochen Yu Categories: cs.CL, cs.AI, cs.LG

The Gap

World models serve as the mental simulation engines of intelligence: they anticipate future states, forecast the consequences of candidate interventions, and allow agents to plan without burning real-world trials.

Yet existing predictive architectures remain fiercely siloed. Video models (e.g. V-JEPA, Sora) operate over spatiotemporal visual patches; molecular simulators run specialized graph neural networks; climate surrogates rely on spherical harmonics; and biological systems depend on gene-regulatory graphs.

The underlying question has sat unanswered: Can a single, unified predictive learning principle govern world modeling across radically heterogeneous dynamical systems?

Vanilla Joint-Embedding Predictive Architectures (JEPAs) eliminate pixel-level generative waste by predicting in latent abstract feature space. However, when applied to complex physical and scientific systems, vanilla JEPAs suffer from factor entanglement: predictive latents lump fast vs. slow modes, invariant versus intervened factors, and global conservation laws into a single undifferentiated vector. This entanglement leads to cascading error drift over long rollouts and renders targeted interventions unpredictable.

   THE CHALLENGE: FRAGMENTED DYNAMICS PREDICTION

   Visual Worlds      Molecular Dynamics     Biological Cells      Celestial Orbits
   (Pixels/Patches)   (Atoms & Forces)       (Gene Regulatory)     (Gravity/Kepler)
         \                  |                      |                     /
          \                 |                      |                    /
           v                v                      v                   v
     [ Domain-Specific Models ] [ Custom Simulators ] [ Statistical Trajectories ]
                             |
                             v
   UNIFIED QUESTION: Can one predictive formulation model them all?
                             |
                             v
   LIMITATION OF VANILLA JEPA:
     Single monolithic latent space -> Fast/slow modes entangled
     -> Long-horizon rollout collapses -> Interventions unpredictable
                             |
                             v
   SOLUTION: JEPA-ANYTHING + Orthogonal Predictive Factorization (OPF)
     Latent Target Decomposed into Orthogonal Dedicated Pathways

The Increment

One sentence: JEPA-Anything extends Joint-Embedding Predictive Architectures into a domain-agnostic foundation via Orthogonal Predictive Factorization (OPF), decomposing complex state transitions into complementary latent factors that yield lower rollout error across 7 distinct scientific domains and accurately recover Keplerian orbital laws.

Core Mechanism

Rather than predicting the entire subsequent state embedding as a monolithic vector $z_{t+1}$, Orthogonal Predictive Factorization (OPF) dissects the target latent manifold into mutually complementary sub-spaces:

1. Orthogonal Pathway Decomposition: The target state representation is factored into orthogonal subspaces: $z = \bigoplus_{k=1}^K z^{(k)}, \quad \text{subject to } \langle z^{(i)}, z^{(j)} \rangle \approx 0 \quad (\forall i \neq j)$ Each subspace captures distinct dynamics: e.g., conserved geometric invariants, high-frequency localized vibrations, and controllable action-sensitive transitions.
2. Dedicated Predictor Branches: Each factor is predicted along its own parameterized path, conditioned on prior history and external interventions.
3. Recombinant Co-Attention: The factorized predictions are recombined through a shared interaction layer that enforces joint consistency without allowing cross-factor collapse.

   ORTHOGONAL PREDICTIVE FACTORIZATION (OPF)

   Input Context (t) -----------------------------> Factor Predictor
          |                                               |
          v                                               v
   +---------------+                              +---------------+
   | Shared Trunk  |                              | Dedicated     |
   | State Encoder |                              | Pathways      |
   +---------------+                              +---------------+
          |                                        |      |      |
          v                                        v      v      v
   Latent Factors: [ Conservation ] [ Fast Fluctuation ] [ Action Shift ]
          \                  |                  /
           \                 |                 /
            v                v                v
      +----------------------------------------------+
      |        Orthogonal Alignment Constraint       |
      +----------------------------------------------+
                             |
                             v
               Predicted Next Latent z_{t+1}

The structural metaphor is a symphony orchestra’s multi-track mixing console. A vanilla predictive model is like recording an entire orchestra with a single cheap omnidirectional microphone. If you try to predict how the piece will sound two measures later, the booming timpani drowns out the quiet oboe melody, and you cannot adjust or “intervene” on the violin section without distorting the whole track. JEPA-Anything equips the conductor with isolated directional microphones (dedicated orthogonal pathways) for each instrumental section—strings, brass, woodwinds, percussion. Each track is predicted along its own acoustic logic (timpani follow rhythm; flutes follow melodic phrasing) and then recombined at the master mixing board. Because the tracks are cleanly isolated, you can change a single violin note (an intervention) without breaking the tempo of the drums.

Key Concepts

• Joint-Embedding Predictive Architecture (JEPA): A paradigm proposed by Yann LeCun that learns representations by predicting representations of masked or future states rather than reconstructing raw pixels or sensor noise.
• Orthogonal Predictive Factorization (OPF): Decomposing high-dimensional dynamical transitions into linearly or geometrically independent latent sub-factors to avoid feature entanglement during multi-step auto-regressive rollout.
• Intervention Generalization: The ability of a world model to accurately predict system response when a variable is actively manipulated (counterfactual/do-calculus), tested from classic Atari games to biological organoids.

Framework Shift

Before (Domain-Specific or Entangled Latent Models):
  Domain X -> Specialized Architecture -> Monolithic Latent Transition
  -> Invariant properties bleed into transient fluctuations
  -> Multi-step rollouts drift into nonsense after 10-20 steps
  -> Interventions perturb un-targeted physical variables

After (Domain-Agnostic Factorized Dynamics):
  Any Dynamical System -> Generic Transformer Trunk -> Orthogonal Factor Pathways
  -> Conserved quantities preserved separately from high-frequency noise
  -> Stable 100-step molecular dynamics rollouts
  -> Verified physical constants: fitted Kepler exponent of -1.4991 (theory: -1.5)

From “designing custom neural PDE simulators or video generators for each scientific vertical,” the core shift is unifying heterogeneous dynamics under an orthogonal, factorized joint-embedding objective.

Expert Assessment

Problem choice: Audacious and fundamental. Tackling world modeling not as a narrow visual forecasting trick but as a universal scientific representation principle across physical, chemical, and biological dynamics is ambitious.

Method maturity: The mathematical framework of orthogonal predictive factorization is well-motivated. It provides a formal answer to the representation collapse and drift issues that have limited JEPAs in precision-critical domains.

Experimental integrity: The empirical evidence is exceptionally broad:

• Coverage: 7 disparate domains (vision, molecular dynamics, clinical data, fluid/physical fields, robotics control, weather, cell biology).
• Hard Benchmarks: 100-step molecular rollouts without divergence; 34.8% error reduction on Interventional Pong; over 1,000 clinical forecasting events.
• Physical Grounding: Recovers Kepler’s third law scaling exponent ($T \propto a^{3/2} \implies \text{slope } -1.5$) with empirical fit of -1.4991.
• Wet-lab Validation: Biological intervention predictions confirmed in vitro (organoids, cell co-cultures) and in vivo (tumor fragments in mice).

Writing quality: Exemplary clarity. The paper avoids over-claiming, systematically contrasts against matched vanilla JEPA baselines, and open-sources reproducible code.

Verdict: strong accept — A landmark contribution that pushes self-supervised predictive representation learning into the territory of general scientific discovery.

Takeaways

• Stop predicting entangled monolithic latent vectors for complex dynamical systems; decompose your target state into orthogonal factor pathways.
• JEPAs can model far more than video: any system governed by time-series transitions (molecules, medical logs, planetary physics) can benefit from non-generative representation prediction.
• Validate world models not merely on reconstruction loss, but on multi-step rollout stability, conservation law retention, and causal intervention accuracy.