LucidDreaming: Controllable Object-Centric 3D Generation

TL;DR

LucidDreaming introduces a pipeline that enables precise spatial and numerical control over 3D object generation from text prompts or 3D bounding boxes, leveraging LLMs for improved accuracy and flexibility.

Contribution

The paper presents a novel method that uses LLMs and new sampling techniques to achieve controllable 3D generation, including object placement and scene editing, with broad applicability.

Findings

Achieves high precision in object placement within 3D scenes.

Demonstrates compatibility with multiple 3D generation frameworks.

Provides a new dataset for benchmarking 3D spatial controllability.

Abstract

With the recent development of generative models, Text-to-3D generations have also seen significant growth, opening a door for creating video-game 3D assets from a more general public. Nonetheless, people without any professional 3D editing experience would find it hard to achieve precise control over the 3D generation, especially if there are multiple objects in the prompt, as using text to control often leads to missing objects and imprecise locations. In this paper, we present LucidDreaming as an effective pipeline capable of spatial and numerical control over 3D generation from only textual prompt commands or 3D bounding boxes. Specifically, our research demonstrates that Large Language Models (LLMs) possess 3D spatial awareness and can effectively translate textual 3D information into precise 3D bounding boxes. We leverage LLMs to get individual object information and their 3D bounding boxes as the initial step of our process. Then with the bounding boxes, We further propose clipped ray sampling and object-centric density blob bias to generate 3D objects aligning with the bounding boxes. We show that our method exhibits remarkable adaptability across a spectrum of mainstream Score Distillation Sampling-based 3D generation frameworks and our pipeline can even used to insert objects into an existing NeRF scene. Moreover, we also provide a dataset of prompts with 3D bounding boxes, benchmarking 3D spatial controllability. With extensive qualitative and quantitative experiments, we demonstrate that LucidDreaming achieves superior results in object placement precision and generation fidelity compared to current approaches, while maintaining flexibility and ease of use for non-expert users.