Chat-Edit-3D: Interactive 3D Scene Editing via Text Prompts

TL;DR

CE3D introduces a flexible, dialogue-based 3D scene editing system that leverages large language models and Hash-Atlas representations, enabling seamless integration of various visual models for diverse, interactive editing.

Contribution

The paper presents a novel dialogue-driven 3D editing framework that decouples 2D and 3D processes, allowing flexible integration of multiple visual models without complex fusion.

Findings

Effective multi-model integration for diverse editing effects

Strong scene comprehension and multi-round dialogue capabilities

Decoupled 2D and 3D editing processes

Abstract

Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still exhibit certain shortcomings, hindering their further interactive design. Such schemes typically adhere to fixed input patterns, limiting users' flexibility in text input. Moreover, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose a dialogue-based 3D scene editing approach, termed CE3D, which is centered around a large language model that allows for arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual expert models. Furthermore, we design a scheme utilizing Hash-Atlas to represent 3D scene views, which transfers the editing of 3D scenes onto 2D atlas images. This design achieves complete decoupling between the 2D editing and 3D reconstruction processes, enabling CE3D to flexibly integrate a wide range of existing 2D or 3D visual models without necessitating intricate fusion designs. Experimental results demonstrate that CE3D effectively integrates multiple visual models to achieve diverse editing visual effects, possessing strong scene comprehension and multi-round dialog capabilities. The code is available at https://sk-fun.fun/CE3D.