Deep Single-View 3D Object Reconstruction with Visual Hull Embedding

TL;DR

This paper introduces a deep learning approach that enhances 3D object reconstruction from a single view by integrating probabilistic visual hulls, leading to more detailed and consistent shape reconstructions.

Contribution

It proposes a novel method combining CNN-based shape, pose, and silhouette prediction with visual hull embedding for improved 3D shape refinement.

Findings

Significantly improves shape detail recovery.

Enhances shape consistency with input images.

Effective on both synthetic and real data.

Abstract

3D object reconstruction is a fundamental task of many robotics and AI problems. With the aid of deep convolutional neural networks (CNNs), 3D object reconstruction has witnessed a significant progress in recent years. However, possibly due to the prohibitively high dimension of the 3D object space, the results from deep CNNs are often prone to missing some shape details. In this paper, we present an approach which aims to preserve more shape details and improve the reconstruction quality. The key idea of our method is to leverage object mask and pose estimation from CNNs to assist the 3D shape learning by constructing a probabilistic single-view visual hull inside of the network. Our method works by first predicting a coarse shape as well as the object pose and silhouette using CNNs, followed by a novel 3D refinement CNN which refines the coarse shapes using the constructed probabilistic visual hulls. Experiment on both synthetic data and real images show that embedding a single-view visual hull for shape refinement can significantly improve the reconstruction quality by recovering more shapes details and improving shape consistency with the input image.