From 2D to 3D Cognition: A Brief Survey of General World Models

TL;DR

This survey reviews the transition from 2D to 3D world models in AI, highlighting recent advances, core capabilities, applications, and future challenges in developing more robust 3D cognitive systems.

Contribution

It provides a structured framework categorizing emerging 3D world modeling techniques and clarifies their roles in advancing artificial general intelligence.

Findings

Identification of key technological drivers in 3D world models

Dissection of three core 3D cognitive capabilities

Analysis of applications in AI, autonomous driving, and gaming

Abstract

World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.