Diffusion Models in 3D Vision: A Survey

TL;DR

This survey reviews how diffusion models are increasingly applied to 3D vision tasks, highlighting their mathematical foundations, recent advancements, challenges, and future directions in the field.

Contribution

It provides a comprehensive overview of diffusion models in 3D vision, including mathematical principles, architectural innovations, and key challenges, serving as a foundation for future research.

Findings

Diffusion models improve 3D object generation and reconstruction.

Handling occlusions and point density variations remains challenging.

Potential for large-scale pretraining to enhance 3D diffusion models.

Abstract

In recent years, 3D vision has become a crucial field within computer vision, powering a wide range of applications such as autonomous driving, robotics, augmented reality, and medical imaging. This field relies on accurate perception, understanding, and reconstruction of 3D scenes from 2D images or text data sources. Diffusion models, originally designed for 2D generative tasks, offer the potential for more flexible, probabilistic methods that can better capture the variability and uncertainty present in real-world 3D data. In this paper, we review the state-of-the-art methods that use diffusion models for 3D visual tasks, including but not limited to 3D object generation, shape completion, point-cloud reconstruction, and scene construction. We provide an in-depth discussion of the underlying mathematical principles of diffusion models, outlining their forward and reverse processes, as well as the various architectural advancements that enable these models to work with 3D datasets. We also discuss the key challenges in applying diffusion models to 3D vision, such as handling occlusions and varying point densities, and the computational demands of high-dimensional data. Finally, we discuss potential solutions, including improving computational efficiency, enhancing multimodal fusion, and exploring the use of large-scale pretraining for better generalization across 3D tasks. This paper serves as a foundation for future exploration and development in this rapidly evolving field.