Multi-view 3D Models from Single Images with a Convolutional Network

TL;DR

This paper introduces a convolutional network that infers 3D representations from single images, predicting RGB views and depth maps to reconstruct full 3D models of objects, including real-world images.

Contribution

It is the first to generate complete 3D models from a single image using a convolutional network trained on synthetic data.

Findings

Successfully predicts 3D structures from single images.

Handles cluttered backgrounds and real images.

Generates accurate point clouds and surface meshes.

Abstract

We present a convolutional network capable of inferring a 3D representation of a previously unseen object given a single image of this object. Concretely, the network can predict an RGB image and a depth map of the object as seen from an arbitrary view. Several of these depth maps fused together give a full point cloud of the object. The point cloud can in turn be transformed into a surface mesh. The network is trained on renderings of synthetic 3D models of cars and chairs. It successfully deals with objects on cluttered background and generates reasonable predictions for real images of cars.