From Pixels to States: Why Video Generation Alone Won't Build Game Engines

Paper: 2607.14076 Authors: Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, Kaipeng Zhang Categories: cs.CV

The Gap

There’s been a rush of excitement around using video generative models — diffusion models, autoregressive transformers — as interactive game engines. Projects like GameNGen, DIAMOND, and GameGen-X train models to predict the next video frame conditioned on player input, and the results look stunning in demos. The pitch is seductive: why write rules by hand when a neural net can just learn the world from footage?

But here’s the problem the paper identifies: these approaches fundamentally misunderstand what a game engine actually does. A real game engine (Unity, Unreal, or even a simple ECS architecture) runs a recurrent loop: action → state update → render observation. The “state” is the source of truth — it knows where every object is, what HP the boss has, which doors are locked. Observations are *derived from state, not the other way around. Video-gen models invert this: they try to go directly from pixels to pixels, skipping the state entirely. The consequences are real — actions don’t produce consistent long-horizon effects, game rules aren’t enforced, and there’s no way to “pause and inspect” the world.

This paper’s contribution is twofold: (1) a structured framework that dissects the gap along four dimensions, and (2) a concrete dataset that enables the state-aware direction. It’s a “let’s take stock and chart the path” paper, not a new-model paper.

Video gen models predict pixels   Player actions should have
conditioned on actions             persistent, rule-governed effects
         |                                    |
         v                                    v
  Looks impressive in             But lacks: explicit state,
  short clips                     long-horizon consistency,
         |                        real-time interactivity
         |                                    |
         +---------> GAP <--------------------+
                       |
                       v
        Real engines use: Action -> State -> Observe loop
                       |
                       v
        Paper proposes: 4-dimension framework to diagnose
        which components video-gen approaches are missing
                       |
                       v
        Evidence: 90hr Black Myth:Wukong dataset with
        ground-truth game states enables state-aware training
                       |
                       v
        Conclusion: State-aware models are the path forward;
        current pixel-to-pixel paradigm is structurally limited

The Increment

One sentence: Before this paper, the “video gen as game engine” hype lacked a principled diagnostic framework; after it, we have a 4-dimensional lens to see exactly what’s broken, and a dataset to fix it.

Core Mechanism

This isn’t a methods paper with a novel architecture — it’s a framework paper. The authors organize the design space of interactive world models around four dimensions, each drawn from how conventional game engines work:

Dimension 1: Player Action Control — How does the model receive and interpret user input? Approaches range from direct pixel-level conditioning (keyboard → frame) to learned action embeddings to structured action spaces. The paper argues that structured action representations outperform raw conditioning because they disentangle “what the player wants” from “how the world renders.”

Dimension 2: Game State Dynamics — Does the model maintain any internal state, or is it purely reactive? The authors categorize approaches into stateless (predict next frame from history buffer) vs. state-maintaining (explicit latent state that evolves). This is the crux: without state, there’s no mechanism to enforce rules or track consequences.

Dimension 3: State-Observation Persistence — Can the world be “saved and loaded”? In a real engine, you serialize the state and reconstruct observations later. Video-gen models typically can’t — if you stop generating, the world is lost. The paper examines approaches that attempt to separate state from rendering.

Dimension 4: Real-Time Interactive Generation — Can the model run fast enough to be playable? This involves latency, throughput, and the generation loop architecture (autoregressive vs. diffusion vs. hybrid).

The dataset contribution targets Dimensions 2 and 3 specifically: by collecting ground-truth game states alongside frames from Black Myth: Wukong, they create training data that includes the state information video-gen models normally lack.

                    [Player Input]
                         |
                         v
              +-----------------------+
              | 1. Action Control     |
              | (how input encodes)   |
              +-----------------------+
                         |
                         v
              +-----------------------+
              | 2. State Dynamics     | <--- Game state: the source of truth
              | (state update logic)  |      (HP, positions, inventory, etc.)
              +-----------------------+
                         |
                         v
              +-----------------------+
              | 3. State-Observation  |
              |    Persistence        |
              | (serialize/restore)   |
              +-----------------------+
                         |
                         v
              +-----------------------+
              | 4. Real-Time Gen      |
              | (rendering latency)   |
              +-----------------------+
                         |
                         v
                  [Player Sees Frame]

  Traditional Engine:  all 4 dimensions explicit, rule-based
  Video-Gen Approach:  skips 2, weak on 3, struggles with 4
  This Paper's Lens:   diagnose which dimension each prior work fails on

Structural Metaphor

Think of building a self-driving car simulator.

A traditional game engine is like a real car with a dashboard camera. There’s an actual engine (the physics simulation), an actual steering column (the action interface), and the dashcam just shows you what’s already there (rendering). If you pause, the car is still parked where it was. If you rewind, everything is consistent.

A video-gen-as-game-engine approach is like having a person in a room watching dashcam footage and trying to produce the *next frame of footage based on what steering wheel angle you show them. They’ve seen a lot of driving footage, so they’re pretty good at it for short stretches! But they don’t actually know where the car is on the road. If you ask them “is there a red light ahead?”, they can only guess from visual patterns. If you pause and resume, they’ve lost track. If you turn left for too long, they might eventually render you driving into a lake because they have no persistent model of road geometry.

This paper is essentially saying: “Let’s look at exactly which pieces of the real car are missing from the dashcam-watcher setup.” And then they hand you a dataset that includes not just dashcam footage, but also GPS coordinates, speedometer readings, and engine telemetry — so you could train a model that actually understands the car, not just mimics the camera.

Key Concepts

• State vs. Observation: In a game, the “state” is the complete truth — every enemy’s HP, every item’s position, every quest flag. The “observation” is just what the camera sees: a rendered frame. A real engine computes observation *from state (state → render → pixels). Video-gen models try to go pixels → pixels, skipping the state. This is like trying to predict tomorrow’s weather by looking at a photo of today’s sky instead of reading the barometer, humidity, and wind data. The photo contains weather information, but it’s lossy, ambiguous, and can’t be “run forward” reliably.

• Recurrent Action-State Loop: The fundamental architecture of any game engine: the player acts, the state updates, a new observation is rendered, repeat. Each step is clean and modular. The paper argues this loop is what makes games *feel responsive and consistent. Video-gen models either skip the loop entirely (one-shot prediction) or approximate it with autoregressive frame prediction, which accumulates errors over time like a game of telephone.

• The Data Engine Problem: The paper’s most practical contribution. To train a state-aware world model, you need training data that *includes game states — but most gameplay footage only has pixels. The authors built a data collection pipeline for Black Myth: Wukong that captures frame-aligned player actions, full game state dumps, and rendered observations simultaneously, plus semantic annotations. This is the kind of unglamorous infrastructure work that actually unblocks the next generation of models.

Framework Shift

Before (mainstream approach):              After (this paper):

   [Player Input]                             [Player Input]
        |                                          |
        v                                          v
  +------------+                           +------------+
  | Video Gen  |                           | Action     |
  | Model      |                           | Encoder    |
  |            |                           +------------+
  | (implicit  |                                  |
  |  state in  |                                  v
  |  weights)  |                           +------------+
  |            |                           | Explicit   |
  +------------+                           | Game State |
        |                                  | (latent or |
        v                                  |  structured)|
  [Next Frame]                             +------------+
                                                  |
   Pixel -> Pixel                                v
   (no state, no rules,                    +------------+
    no persistence)                        | State-to-  |
                                           | Observation|
                                           | Renderer   |
                                           +------------+
                                                  |
                                                  v
                                           [Next Frame]

                                           State -> Render -> Pixel
                                           (state persists, rules enforceable)

From implicit-weights-as-state to explicit-state-then-render, the core shift is recognizing that pixel prediction is an observation problem, not a world modeling problem — and you need the world model underneath.

Expert Assessment

Problem choice: This is a real and well-timed gap. The hype around “video gen = game engine” has outpaced the actual capabilities, and nobody had done a rigorous dissection of *why these systems feel brittle in practice. The four-dimension framework is genuinely useful — it gives researchers a vocabulary to say “my method improves on Dimension 2 but not Dimension 3” instead of vague claims. That said, the paper sometimes reads more like a tutorial than a research contribution; the framework is descriptive rather than prescriptive.

Method maturity: This is primarily a survey/taxonomy paper with a dataset contribution, not a novel method paper. The framework is well-organized but not deeply novel — anyone who’s built a game engine would recognize the action-state-observe loop as obvious. The real value is in *applying this lens systematically to the video-gen literature. The dataset is the concrete contribution, and it’s a good one — 90 hours of state-aligned data from a AAA game is non-trivial to collect.

Experimental integrity: There are no model training experiments or benchmarks — this is a framework paper. The dataset is described but not used in any downstream experiments within this paper. This is a limitation: they tell us the data *could enable state-aware training, but don’t show it. Future work, presumably. The dataset quality appears sound based on description, but without experiments, we can’t verify the claims about its utility.

Writing quality: The paper is clearly written with good structure. The four dimensions are well-motivated. However, Section 3 (the taxonomy) sometimes drifts into “listing prior work” rather than making sharp analytical points. The dataset section (Section 4) could be stronger — more details on the annotation pipeline, quality metrics, and failure modes would help. The related work section is oddly thin given this is fundamentally a survey-style paper.

Verdict: weak accept — The framework is useful and the dataset is valuable, but the lack of any experimental validation of the core claims (that state-aware models trained on this data actually outperform pixel-only approaches) leaves the contribution feeling incomplete. It’s a good positioning paper that will get cited, but it doesn’t deliver the payoff it sets up.

Takeaways

1. The 4-dimension diagnostic framework is directly reusable. If you’re evaluating any interactive generation system, ask: how does it handle (a) action encoding, (b) state tracking, (c) state persistence, (d) real-time generation? This applies to robotics simulators, digital twins, and AR/VR worlds too — not just games.

2. The “state-first” design principle transfers broadly. Any domain where people try to predict complex observations directly (video prediction for robotics, weather modeling, traffic simulation) probably has the same issue: you need an intermediate state representation, not end-to-end pixel prediction. The paper makes this argument cleanly.

3. The data collection approach — hooking into a real game engine to extract ground-truth states alongside rendered frames — is a template that other researchers can replicate for other games. The specific pipeline for Black Myth: Wukong is a resource, but the methodology is the transferable asset.

4. Honest limitation: If you’re looking for a new architecture or SOTA results, this paper won’t give you that. Its value is in reframing the conversation. Worth reading for the perspective, not the code.