Diffusion Models Are Real-Time Game Engines

TL;DR

GameNGen is a neural-powered game engine capable of real-time, high-quality, long-trajectory interaction in complex environments, trained on DOOM, and capable of realistic simulation and prediction.

Contribution

This work introduces GameNGen, the first neural game engine that enables real-time interaction and long trajectory simulation using diffusion models trained on gameplay data.

Findings

Runs at 20 FPS on a TPU

Achieves PSNR of 29.4 for next frame prediction

Human raters struggle to distinguish real from generated clips

Abstract

We present GameNGen, the first game engine powered entirely by a neural model that also enables real-time interaction with a complex environment over long trajectories at high quality. When trained on the classic game DOOM, GameNGen extracts gameplay and uses it to generate a playable environment that can interactively simulate new trajectories. GameNGen runs at 20 frames per second on a single TPU and remains stable over extended multi-minute play sessions. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation, even after 5 minutes of auto-regressive generation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations help ensure stable auto-regressive generation over long trajectories, and decoder fine-tuning improves the fidelity of visual details and text.