MVPNet: Multi-View Point Regression Networks for 3D Object Reconstruction from A Single Image

TL;DR

MVPNet introduces a multi-view point regression approach using dense point clouds and a novel geometric loss to improve single-image 3D object reconstruction, outperforming existing methods.

Contribution

The paper presents a new network architecture that regresses multi-view point clouds with a geometric loss, enabling more accurate 3D surface reconstruction from a single image.

Findings

Outperforms state-of-the-art methods on challenging datasets.

Uses a novel geometric loss for better surface discrepancy measurement.

Employs multi-view point clouds aligned on image planes for reconstruction.

Abstract

In this paper, we address the problem of reconstructing an object's surface from a single image using generative networks. First, we represent a 3D surface with an aggregation of dense point clouds from multiple views. Each point cloud is embedded in a regular 2D grid aligned on an image plane of a viewpoint, making the point cloud convolution-favored and ordered so as to fit into deep network architectures. The point clouds can be easily triangulated by exploiting connectivities of the 2D grids to form mesh-based surfaces. Second, we propose an encoder-decoder network that generates such kind of multiple view-dependent point clouds from a single image by regressing their 3D coordinates and visibilities. We also introduce a novel geometric loss that is able to interpret discrepancy over 3D surfaces as opposed to 2D projective planes, resorting to the surface discretization on the constructed meshes. We demonstrate that the multi-view point regression network outperforms state-of-the-art methods with a significant improvement on challenging datasets.