Understanding World or Predicting Future? A Comprehensive Survey of World Models

TL;DR

This survey comprehensively reviews world models, focusing on their roles in understanding and predicting the world, and discusses their applications across various domains and future research challenges.

Contribution

It provides a systematic categorization of world models, highlighting their functions, applications, and future research directions, with a curated list of key papers and code resources.

Findings

World models serve two main functions: understanding and predicting.

Applications span generative games, autonomous driving, robotics, and social simulations.

Key challenges include scalability, generalization, and real-world deployment.

Abstract

The concept of world models has garnered significant attention due to advancements in multimodal large language models such as GPT-4 and video generation models such as Sora, which are central to the pursuit of artificial general intelligence. This survey offers a comprehensive review of the literature on world models. Generally, world models are regarded as tools for either understanding the present state of the world or predicting its future dynamics. This review presents a systematic categorization of world models, emphasizing two primary functions: (1) constructing internal representations to understand the mechanisms of the world, and (2) predicting future states to simulate and guide decision-making. Initially, we examine the current progress in these two categories. We then explore the application of world models in key domains, including generative games, autonomous driving, robotics, and social simulacra, with a focus on how each domain utilizes these aspects. Finally, we outline key challenges and provide insights into potential future research directions. We summarize the representative papers along with their code repositories in https://github.com/tsinghua-fib-lab/World-Model.