Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation

TL;DR

This paper introduces a novel diffusion model that generates colored 3D point clouds by jointly conditioning on sketches and textual descriptions, improving shape accuracy and color assignment.

Contribution

The proposed model uniquely combines sketch and text guidance with staged diffusion for detailed colored point cloud generation, advancing 3D shape synthesis methods.

Findings

Outperforms recent state-of-the-art in point cloud generation

Effectively integrates sketch and text for detailed 3D shape creation

Enables tasks like appearance re-editing and part segmentation

Abstract

Diffusion probabilistic models have achieved remarkable success in text guided image generation. However, generating 3D shapes is still challenging due to the lack of sufficient data containing 3D models along with their descriptions. Moreover, text based descriptions of 3D shapes are inherently ambiguous and lack details. In this paper, we propose a sketch and text guided probabilistic diffusion model for colored point cloud generation that conditions the denoising process jointly with a hand drawn sketch of the object and its textual description. We incrementally diffuse the point coordinates and color values in a joint diffusion process to reach a Gaussian distribution. Colored point cloud generation thus amounts to learning the reverse diffusion process, conditioned by the sketch and text, to iteratively recover the desired shape and color. Specifically, to learn effective sketch-text embedding, our model adaptively aggregates the joint embedding of text prompt and the sketch based on a capsule attention network. Our model uses staged diffusion to generate the shape and then assign colors to different parts conditioned on the appearance prompt while preserving precise shapes from the first stage. This gives our model the flexibility to extend to multiple tasks, such as appearance re-editing and part segmentation. Experimental results demonstrate that our model outperforms recent state-of-the-art in point cloud generation.