Predicting Complete 3D Models of Indoor Scenes

TL;DR

This paper presents a method to interpret indoor scenes from a single RGBD image by generating complete 3D models of walls and objects, aiding robotics and visual reasoning.

Contribution

It introduces a data-driven approach for complete 3D scene parsing from a single image, including occluded regions, using CAD-like models and Manhattan structure constraints.

Findings

Effective scene interpretation on NYU v2 dataset

Successful modeling of both visible and occluded scene parts

Potential for improved robotics and visual reasoning applications

Abstract

One major goal of vision is to infer physical models of objects, surfaces, and their layout from sensors. In this paper, we aim to interpret indoor scenes from one RGBD image. Our representation encodes the layout of walls, which must conform to a Manhattan structure but is otherwise flexible, and the layout and extent of objects, modeled with CAD-like 3D shapes. We represent both the visible and occluded portions of the scene, producing a complete 3D parse. Such a scene interpretation is useful for robotics and visual reasoning, but difficult to produce due to the well-known challenge of segmentation, the high degree of occlusion, and the diversity of objects in indoor scene. We take a data-driven approach, generating sets of potential object regions, matching to regions in training images, and transferring and aligning associated 3D models while encouraging fit to observations and overall consistency. We demonstrate encouraging results on the NYU v2 dataset and highlight a variety of interesting directions for future work.