Reconstructing 4D Spatial Intelligence: A Survey

TL;DR

This survey organizes 4D spatial reconstruction methods into five hierarchical levels, providing a comprehensive analysis of recent advances, challenges, and future directions in 4D scene understanding from visual data.

Contribution

It introduces a new hierarchical framework for 4D scene reconstruction, offering a structured perspective that encompasses recent progress and identifies key challenges.

Findings

Organized existing methods into five progressive levels of 4D spatial intelligence.

Identified key challenges and promising directions at each level.

Provided an up-to-date resource with a project page for ongoing developments.

Abstract

Reconstructing 4D spatial intelligence from visual observations has long been a central yet challenging task in computer vision, with broad real-world applications. These range from entertainment domains like movies, where the focus is often on reconstructing fundamental visual elements, to embodied AI, which emphasizes interaction modeling and physical realism. Fueled by rapid advances in 3D representations and deep learning architectures, the field has evolved quickly, outpacing the scope of previous surveys. Additionally, existing surveys rarely offer a comprehensive analysis of the hierarchical structure of 4D scene reconstruction. To address this gap, we present a new perspective that organizes existing methods into five progressive levels of 4D spatial intelligence: (1) Level 1 -- reconstruction of low-level 3D attributes (e.g., depth, pose, and point maps); (2) Level 2 -- reconstruction of 3D scene components (e.g., objects, humans, structures); (3) Level 3 -- reconstruction of 4D dynamic scenes; (4) Level 4 -- modeling of interactions among scene components; and (5) Level 5 -- incorporation of physical laws and constraints. We conclude the survey by discussing the key challenges at each level and highlighting promising directions for advancing toward even richer levels of 4D spatial intelligence. To track ongoing developments, we maintain an up-to-date project page: https://github.com/yukangcao/Awesome-4D-Spatial-Intelligence.