LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Model

TL;DR

LLplace is a novel 3D indoor scene layout generator and editor that uses a fine-tuned open-source LLM, enabling dynamic scene editing and layout generation solely based on user inputs without relying on spatial priors.

Contribution

The paper introduces LLplace, a lightweight fine-tuned Llama3-based model that enables efficient 3D indoor scene design and editing through dialogue, bypassing the need for spatial priors and extensive prompt engineering.

Findings

Effective generation of 3D indoor layouts from user inputs.

Supports dynamic scene editing via dialogue commands.

Outperforms existing methods in quality and flexibility.

Abstract

Designing 3D indoor layouts is a crucial task with significant applications in virtual reality, interior design, and automated space planning. Existing methods for 3D layout design either rely on diffusion models, which utilize spatial relationship priors, or heavily leverage the inferential capabilities of proprietary Large Language Models (LLMs), which require extensive prompt engineering and in-context exemplars via black-box trials. These methods often face limitations in generalization and dynamic scene editing. In this paper, we introduce LLplace, a novel 3D indoor scene layout designer based on lightweight fine-tuned open-source LLM Llama3. LLplace circumvents the need for spatial relationship priors and in-context exemplars, enabling efficient and credible room layout generation based solely on user inputs specifying the room type and desired objects. We curated a new dialogue dataset based on the 3D-Front dataset, expanding the original data volume and incorporating dialogue data for adding and removing objects. This dataset can enhance the LLM's spatial understanding. Furthermore, through dialogue, LLplace activates the LLM's capability to understand 3D layouts and perform dynamic scene editing, enabling the addition and removal of objects. Our approach demonstrates that LLplace can effectively generate and edit 3D indoor layouts interactively and outperform existing methods in delivering high-quality 3D design solutions. Code and dataset will be released.