DiffS-NOCS: 3D Point Cloud Reconstruction through Coloring Sketches to NOCS Maps Using Diffusion Models

TL;DR

DiffS-NOCS introduces a diffusion-based method that reconstructs 3D point clouds from sketches by generating NOCS maps with multi-view consistency, enabling controllable and detailed 3D reconstructions.

Contribution

The paper presents a novel diffusion model approach with multi-view aggregation and viewpoint encoding for sketch-to-3D reconstruction, improving accuracy and control.

Findings

Achieves high-quality 3D reconstructions from sketches.

Demonstrates improved multi-view consistency in NOCS maps.

Enables controllable 3D point cloud generation.

Abstract

Reconstructing a 3D point cloud from a given conditional sketch is challenging. Existing methods often work directly in 3D space, but domain variability and difficulty in reconstructing accurate 3D structures from 2D sketches remain significant obstacles. Moreover, ideal models should also accept prompts for control, in addition with the sparse sketch, posing challenges in multi-modal fusion. We propose DiffS-NOCS (Diffusion-based Sketch-to-NOCS Map), which leverages ControlNet with a modified multi-view decoder to generate NOCS maps with embedded 3D structure and position information in 2D space from sketches. The 3D point cloud is reconstructed by combining multiple NOCS maps from different views. To enhance sketch understanding, we integrate a viewpoint encoder for extracting viewpoint features. Additionally, we design a feature-level multi-view aggregation network as the denoising module, facilitating cross-view information exchange and improving 3D consistency in NOCS map generation. Experiments on ShapeNet demonstrate that DiffS-NOCS achieves controllable and fine-grained point cloud reconstruction aligned with sketches.