iControl3D: An Interactive System for Controllable 3D Scene Generation

TL;DR

iControl3D is an interactive system that allows users to generate and customize 3D scenes with precise control, leveraging 3D meshes, depth alignment, and neural rendering for improved scene creation.

Contribution

The paper introduces iControl3D, a novel interactive system that combines mesh-based scene editing, depth alignment, and neural rendering to enable controllable 3D scene generation.

Findings

Effective integration of 2D diffusion images into 3D scenes

Boundary-aware depth alignment improves mesh fusion

Neural rendering facilitates online scene navigation

Abstract

3D content creation has long been a complex and time-consuming process, often requiring specialized skills and resources. While recent advancements have allowed for text-guided 3D object and scene generation, they still fall short of providing sufficient control over the generation process, leading to a gap between the user's creative vision and the generated results. In this paper, we present iControl3D, a novel interactive system that empowers users to generate and render customizable 3D scenes with precise control. To this end, a 3D creator interface has been developed to provide users with fine-grained control over the creation process. Technically, we leverage 3D meshes as an intermediary proxy to iteratively merge individual 2D diffusion-generated images into a cohesive and unified 3D scene representation. To ensure seamless integration of 3D meshes, we propose to perform boundary-aware depth alignment before fusing the newly generated mesh with the existing one in 3D space. Additionally, to effectively manage depth discrepancies between remote content and foreground, we propose to model remote content separately with an environment map instead of 3D meshes. Finally, our neural rendering interface enables users to build a radiance field of their scene online and navigate the entire scene. Extensive experiments have been conducted to demonstrate the effectiveness of our system. The code will be made available at https://github.com/xingyi-li/iControl3D.