Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene

TL;DR

This paper presents a CNN-based method to extract 3D scene structure, including layout and object pose, from a single 2D image, advancing scene understanding in indoor environments.

Contribution

It introduces a novel approach for factorizing 3D scene elements from 2D images and provides extensive benchmarking on indoor scene datasets.

Findings

Successful inference of 3D layout and object pose from 2D images

Quantitative and qualitative validation of the proposed representation

Insights into practical design choices for scene factorization

Abstract

The goal of this paper is to take a single 2D image of a scene and recover the 3D structure in terms of a small set of factors: a layout representing the enclosing surfaces as well as a set of objects represented in terms of shape and pose. We propose a convolutional neural network-based approach to predict this representation and benchmark it on a large dataset of indoor scenes. Our experiments evaluate a number of practical design questions, demonstrate that we can infer this representation, and quantitatively and qualitatively demonstrate its merits compared to alternate representations.