Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model

TL;DR

This paper introduces a hierarchical approach to indoor scene synthesis using pre-trained large language models, enabling the generation of realistic, diverse, and user-aligned 3D indoor scenes through structured descriptions and layout optimization.

Contribution

The paper proposes a hierarchy-aware network and divide-and-conquer optimization for feasible scene layout generation from LLM outputs, improving realism and generalization in open-vocabulary indoor scene synthesis.

Findings

Generated scene layouts are more reasonable and aligned with user requirements.

Hierarchical structure improves object placement consistency and layout feasibility.

The approach outperforms existing methods in qualitative and quantitative evaluations.

Abstract

Indoor scene synthesis aims to automatically produce plausible, realistic and diverse 3D indoor scenes, especially given arbitrary user requirements. Recently, the promising generalization ability of pre-trained large language models (LLM) assist in open-vocabulary indoor scene synthesis. However, the challenge lies in converting the LLM-generated outputs into reasonable and physically feasible scene layouts. In this paper, we propose to generate hierarchically structured scene descriptions with LLM and then compute the scene layouts. Specifically, we train a hierarchy-aware network to infer the fine-grained relative positions between objects and design a divide-and-conquer optimization to solve for scene layouts. The advantages of using hierarchically structured scene representation are two-fold. First, the hierarchical structure provides a rough grounding for object arrangement, which alleviates contradictory placements with dense relations and enhances the generalization ability of the network to infer fine-grained placements. Second, it naturally supports the divide-and-conquer optimization, by first arranging the sub-scenes and then the entire scene, to more effectively solve for a feasible layout. We conduct extensive comparison experiments and ablation studies with both qualitative and quantitative evaluations to validate the effectiveness of our key designs with the hierarchically structured scene representation. Our approach can generate more reasonable scene layouts while better aligned with the user requirements and LLM descriptions. We also present open-vocabulary scene synthesis and interactive scene design results to show the strength of our approach in the applications.