PointDreamer: Zero-shot 3D Textured Mesh Reconstruction from Colored Point Cloud

TL;DR

PointDreamer introduces a zero-shot framework that leverages 2D diffusion models and a novel unprojection pipeline to produce high-quality textured 3D meshes from colored point clouds, overcoming previous limitations in texture quality and data requirements.

Contribution

It adapts 2D diffusion models for 3D textured mesh reconstruction from point clouds using a project-inpaint-unproject pipeline and proposes a new unprojection strategy to reduce artifacts.

Findings

Achieves 30% improvement in LPIPS score over previous methods.

Demonstrates robustness to noisy, sparse, and incomplete data.

Outperforms existing methods in qualitative and quantitative evaluations.

Abstract

Faithfully reconstructing textured meshes is crucial for many applications. Compared to text or image modalities, leveraging 3D colored point clouds as input (colored-PC-to-mesh) offers inherent advantages in comprehensively and precisely replicating the target object's 360{\deg} characteristics. While most existing colored-PC-to-mesh methods suffer from blurry textures or require hard-to-acquire 3D training data, we propose PointDreamer, a novel framework that harnesses 2D diffusion prior for superior texture quality. Crucially, unlike prior 2D-diffusion-for-3D works driven by text or image inputs, PointDreamer successfully adapts 2D diffusion models to 3D point cloud data by a novel project-inpaint-unproject pipeline. Specifically, it first projects the point cloud into sparse 2D images and then performs diffusion-based inpainting. After that, diverging from most existing 3D reconstruction or generation approaches that predict texture in 3D/UV space thus often yielding blurry texture, PointDreamer achieves high-quality texture by directly unprojecting the inpainted 2D images to the 3D mesh. Furthermore, we identify for the first time a typical kind of unprojection artifact appearing in occlusion borders, which is common in other multiview-image-to-3D pipelines but less-explored. To address this, we propose a novel solution named the Non-Border-First (NBF) unprojection strategy. Extensive qualitative and quantitative experiments on various synthetic and real-scanned datasets demonstrate that PointDreamer, though zero-shot, exhibits SoTA performance (30% improvement on LPIPS score from 0.118 to 0.068), and is robust to noisy, sparse, or even incomplete input data. Code at: https://github.com/YuQiao0303/PointDreamer.