Solaris: Building a Multiplayer Video World Model in Minecraft

Paper: 2602.22208 Authors: Georgy Savva, Oscar Michel, Daohan Lu, Suppakit Waiwitlikhit, Timothy Meehan, Dhairya Mishra, Srivats Poddar, Jack Lu, Saining Xie Categories: cs.CV

Abstract

Solaris represents a significant advancement in video world modeling by extending beyond single-agent perspectives to capture multi-agent interactions. The model simulates consistent multi-view observations in multiplayer environments, specifically demonstrated in Minecraft. The research introduces a comprehensive data collection system that enables robust, continuous, and automated capture of synchronized videos and actions from multiple coordinated agents. Trained on 12.64 million multiplayer frames, Solaris employs a staged training pipeline that progressively transitions from single-player to multiplayer modeling, incorporating bidirectional, causal, and Self Forcing training techniques. The final stage introduces Checkpointed Self Forcing, a memory-efficient variant enabling longer-horizon teacher models. The work establishes a new evaluation framework covering multiplayer movement, memory, grounding, building, and view consistency.

Key Contributions

• First multiplayer video world model capable of simulating consistent multi-view observations across multiple agents
• Novel multiplayer data collection system supporting coordinated multi-agent interaction with synchronized video and action capture
• Large-scale dataset of 12.64 million multiplayer frames from Minecraft gameplay
• Comprehensive evaluation framework for assessing multiplayer capabilities: movement, memory, grounding, building, and view consistency
• Staged training pipeline progressively transitioning from single-player to multiplayer modeling
• Checkpointed Self Forcing technique for memory-efficient training with longer-horizon teachers
• Open-source release of both the data collection system and trained models

Technical Methodology

The Solaris architecture addresses the fundamental challenge of modeling multi-agent interactions through a carefully designed training pipeline. Unlike traditional video world models that operate from a single agent’s perspective, Solaris must maintain consistency across multiple viewpoints while capturing the complex dynamics of agent interactions.

The data collection system represents a critical innovation. Built on top of Minecraft, it enables automated, continuous data gathering from multiple agents simultaneously. The system captures synchronized video frames and action sequences, ensuring temporal alignment across all agent perspectives. This synchronization is essential for training a model that can generate coherent multi-view predictions.

The training methodology employs a three-stage approach. Initially, the model learns from single-player data to establish basic world dynamics and visual generation capabilities. The second stage introduces multi-agent scenarios but maintains simpler interaction patterns. The final stage focuses on complex multiplayer interactions, where Checkpointed Self Forcing becomes crucial. This technique addresses the memory constraints of Self Forcing training by checkpointing intermediate states, allowing the model to learn from longer temporal horizons without excessive memory consumption.

The model architecture itself must handle multiple input streams (one per agent) and generate corresponding output predictions. This requires careful attention to how information flows between agent representations and how the model maintains consistency across different viewpoints of the same scene.

Evaluation Framework and Results

The evaluation framework introduces five key dimensions for assessing multiplayer world model performance. Movement evaluation tests whether the model accurately predicts agent locomotion and navigation in response to actions. Memory evaluation assesses the model’s ability to maintain consistent world state over extended sequences. Grounding evaluation measures how well the model connects actions to their visual consequences. Building evaluation specifically tests construction and modification of the environment, a core Minecraft mechanic. View consistency evaluation ensures that predictions from different agent perspectives remain coherent with each other.

Results demonstrate that Solaris outperforms existing baselines across these metrics. The staged training approach proves essential, with models trained directly on multiplayer data showing inferior performance compared to those following the progressive curriculum. Checkpointed Self Forcing provides measurable improvements in temporal consistency, particularly for longer prediction horizons.

The model shows strong performance in maintaining view consistency, a particularly challenging aspect of multiplayer modeling. When multiple agents observe the same scene from different angles, Solaris generates predictions that remain geometrically and semantically consistent across viewpoints. This capability suggests the model has learned meaningful 3D scene representations rather than merely memorizing 2D patterns.

Implications and Future Directions

Solaris opens new possibilities for multi-agent AI systems and simulation. By demonstrating that video world models can scale to multiplayer scenarios, the work suggests pathways toward more sophisticated agent training environments. Future embodied AI systems could leverage such models for planning, exploration, and learning in complex multi-agent settings.

The open-source release of both the data collection system and trained models lowers barriers for researchers exploring multi-agent world modeling. The Minecraft platform provides a rich, controllable environment for experimentation, and the data collection infrastructure can be adapted for various research questions.

Several directions for future work emerge naturally. Scaling to more agents simultaneously, extending to other game environments or real-world scenarios, and improving the efficiency of multi-view generation all represent important challenges. Additionally, integrating language conditioning could enable more natural human-AI interaction in these multiplayer environments.

The work also raises questions about the relationship between world models and agent policies. While Solaris focuses on prediction, combining it with policy learning could yield agents that better anticipate and respond to multi-agent dynamics.

Takeaways

1. Solaris is the first video world model designed specifically for multiplayer scenarios, capable of generating consistent multi-view observations across multiple agents.

2. The staged training pipeline, progressing from single-player to complex multiplayer scenarios, proves essential for learning robust multi-agent dynamics.

3. Checkpointed Self Forcing enables memory-efficient training with longer-horizon teachers, improving temporal consistency in predictions.

4. The comprehensive evaluation framework covering movement, memory, grounding, building, and view consistency provides a rigorous benchmark for future multiplayer world models.

5. Open-sourcing the data collection system and models establishes infrastructure for advancing multi-agent AI research in interactive environments.