Learning to Simulate Dynamic Environments with GameGAN

TL;DR

GameGAN is a generative model that learns to visually imitate dynamic environments from gameplay data, enabling realistic simulation, environment mapping, and component swapping for game development and AI applications.

Contribution

The paper introduces GameGAN, a novel generative adversarial network that learns to simulate game environments from screen and action data, incorporating memory and disentanglement for interpretability.

Findings

GameGAN accurately reproduces game visuals based on actions.

The model builds an internal map of the environment for consistent navigation.

Disentangling static and dynamic elements improves interpretability.

Abstract

Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN "renders" the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist.