3D Reconstruction of Novel Object Shapes from Single Images

TL;DR

This paper introduces SDFNet, a novel architecture for single-image 3D shape reconstruction that outperforms existing methods on both seen and unseen objects, with extensive analysis of factors affecting performance.

Contribution

Proposes SDFNet, a new model architecture for 3D reconstruction from a single image, and provides the first large-scale evaluation on unseen object shapes.

Findings

SDFNet achieves state-of-the-art results on seen and unseen objects.

Extensive analysis of architecture and training factors influencing performance.

First large-scale evaluation of single-image shape reconstruction on unseen objects.

Abstract

Accurately predicting the 3D shape of any arbitrary object in any pose from a single image is a key goal of computer vision research. This is challenging as it requires a model to learn a representation that can infer both the visible and occluded portions of any object using a limited training set. A training set that covers all possible object shapes is inherently infeasible. Such learning-based approaches are inherently vulnerable to overfitting, and successfully implementing them is a function of both the architecture design and the training approach. We present an extensive investigation of factors specific to architecture design, training, experiment design, and evaluation that influence reconstruction performance and measurement. We show that our proposed SDFNet achieves state-of-the-art performance on seen and unseen shapes relative to existing methods GenRe and OccNet. We provide the first large-scale evaluation of single image shape reconstruction to unseen objects. The source code, data and trained models can be found on https://github.com/rehg-lab/3DShapeGen.