Matrix-Game: Interactive World Foundation Model

TL;DR

Matrix-Game is a large-scale, controllable, interactive world generation model for Minecraft, trained with a two-stage pipeline and evaluated using a new comprehensive benchmark, outperforming previous models in quality and controllability.

Contribution

The paper introduces Matrix-Game, a novel interactive world foundation model with a large dataset and a new benchmark, advancing controllability and physical understanding in game world generation.

Findings

Matrix-Game outperforms prior models in all benchmark metrics.

It achieves high visual quality and temporal coherence.

Human evaluations favor Matrix-Game's realism and controllability.

Abstract

We introduce Matrix-Game, an interactive world foundation model for controllable game world generation. Matrix-Game is trained using a two-stage pipeline that first performs large-scale unlabeled pretraining for environment understanding, followed by action-labeled training for interactive video generation. To support this, we curate Matrix-Game-MC, a comprehensive Minecraft dataset comprising over 2,700 hours of unlabeled gameplay video clips and over 1,000 hours of high-quality labeled clips with fine-grained keyboard and mouse action annotations. Our model adopts a controllable image-to-world generation paradigm, conditioned on a reference image, motion context, and user actions. With over 17 billion parameters, Matrix-Game enables precise control over character actions and camera movements, while maintaining high visual quality and temporal coherence. To evaluate performance, we develop GameWorld Score, a unified benchmark measuring visual quality, temporal quality, action controllability, and physical rule understanding for Minecraft world generation. Extensive experiments show that Matrix-Game consistently outperforms prior open-source Minecraft world models (including Oasis and MineWorld) across all metrics, with particularly strong gains in controllability and physical consistency. Double-blind human evaluations further confirm the superiority of Matrix-Game, highlighting its ability to generate perceptually realistic and precisely controllable videos across diverse game scenarios. To facilitate future research on interactive image-to-world generation, we will open-source the Matrix-Game model weights and the GameWorld Score benchmark at https://github.com/SkyworkAI/Matrix-Game.