GEN3D: Generating Domain-Free 3D Scenes from a Single Image

TL;DR

Gen3d is a novel method that generates high-quality, diverse 3D scenes from a single image, overcoming the limitations of multi-view dependence and advancing applications in embodied AI and world modeling.

Contribution

It introduces a new approach for single-image 3D scene generation that maintains and expands a world model, utilizing Gaussian splatting for high-fidelity scene synthesis.

Findings

Strong generalization across diverse datasets

Superior performance in novel view synthesis

Effective 3D scene generation from minimal input

Abstract

Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. Additionally, 3D scene generation is vital for advancing embodied AI and world models, which depend on diverse, high-quality scenes for learning and evaluation. In this work, we propose Gen3d, a novel method for generation of high-quality, wide-scope, and generic 3D scenes from a single image. After the initial point cloud is created by lifting the RGBD image, Gen3d maintains and expands its world model. The 3D scene is finalized through optimizing a Gaussian splatting representation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in generating a world model and Synthesizing high-fidelity and consistent novel views.