PCDreamer: Point Cloud Completion Through Multi-view Diffusion Priors

TL;DR

PCDreamer introduces a shape completion method leveraging multi-view diffusion priors to generate and fuse shape cues from images and point clouds, achieving superior detail recovery without requiring paired image-point cloud data.

Contribution

The paper proposes a novel point cloud completion approach using multi-view diffusion priors and a shape fusion and consolidation framework, avoiding the need for paired image and point cloud data.

Findings

Outperforms existing methods in recovering fine details.

Effectively utilizes multi-view diffusion priors for shape generation.

Demonstrates superior shape completion accuracy in experiments.

Abstract

This paper presents PCDreamer, a novel method for point cloud completion. Traditional methods typically extract features from partial point clouds to predict missing regions, but the large solution space often leads to unsatisfactory results. More recent approaches have started to use images as extra guidance, effectively improving performance, but obtaining paired data of images and partial point clouds is challenging in practice. To overcome these limitations, we harness the relatively view-consistent multi-view diffusion priors within large models, to generate novel views of the desired shape. The resulting image set encodes both global and local shape cues, which are especially beneficial for shape completion. To fully exploit the priors, we have designed a shape fusion module for producing an initial complete shape from multi-modality input (i.e.,, images and point clouds), and a follow-up shape consolidation module to obtain the final complete shape by discarding unreliable points introduced by the inconsistency from diffusion priors. Extensive experimental results demonstrate our superior performance, especially in recovering fine details.