Blended Point Cloud Diffusion for Localized Text-guided Shape Editing

TL;DR

This paper introduces a novel inpainting-based point cloud diffusion framework for localized, text-guided 3D shape editing that preserves global shape coherence and identity without requiring inversion.

Contribution

It proposes a new diffusion-based editing method with a coordinate blending algorithm for fine-grained, localized shape modifications guided by natural language.

Findings

Outperforms existing methods in shape fidelity and text adherence

Effectively preserves global shape coherence during editing

Enables localized edits without expensive inversion processes

Abstract

Natural language offers a highly intuitive interface for enabling localized fine-grained edits of 3D shapes. However, prior works face challenges in preserving global coherence while locally modifying the input 3D shape. In this work, we introduce an inpainting-based framework for editing shapes represented as point clouds. Our approach leverages foundation 3D diffusion models for achieving localized shape edits, adding structural guidance in the form of a partial conditional shape, ensuring that other regions correctly preserve the shape's identity. Furthermore, to encourage identity preservation also within the local edited region, we propose an inference-time coordinate blending algorithm which balances reconstruction of the full shape with inpainting at a progression of noise levels during the inference process. Our coordinate blending algorithm seamlessly blends the original shape with its edited version, enabling a fine-grained editing of 3D shapes, all while circumventing the need for computationally expensive and often inaccurate inversion. Extensive experiments show that our method outperforms alternative techniques across a wide range of metrics that evaluate both fidelity to the original shape and also adherence to the textual description.