Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation
Zijie Wu, Yaonan Wang, Mingtao Feng, He Xie, Ajmal Mian

TL;DR
This paper introduces a novel diffusion model that generates colored 3D point clouds by jointly conditioning on sketches and textual descriptions, improving shape accuracy and color assignment.
Contribution
The proposed model uniquely combines sketch and text guidance with staged diffusion for detailed colored point cloud generation, advancing 3D shape synthesis methods.
Findings
Outperforms recent state-of-the-art in point cloud generation
Effectively integrates sketch and text for detailed 3D shape creation
Enables tasks like appearance re-editing and part segmentation
Abstract
Diffusion probabilistic models have achieved remarkable success in text guided image generation. However, generating 3D shapes is still challenging due to the lack of sufficient data containing 3D models along with their descriptions. Moreover, text based descriptions of 3D shapes are inherently ambiguous and lack details. In this paper, we propose a sketch and text guided probabilistic diffusion model for colored point cloud generation that conditions the denoising process jointly with a hand drawn sketch of the object and its textual description. We incrementally diffuse the point coordinates and color values in a joint diffusion process to reach a Gaussian distribution. Colored point cloud generation thus amounts to learning the reverse diffusion process, conditioned by the sketch and text, to iteratively recover the desired shape and color. Specifically, to learn effective…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation· youtube
Taxonomy
Topics3D Shape Modeling and Analysis · Human Motion and Animation · Image Processing and 3D Reconstruction
MethodsDiffusion
