SceneNet: Understanding Real World Indoor Scenes With Synthetic Data

TL;DR

This paper demonstrates that synthetic depth data generated from 3D scenes can effectively train models for indoor scene understanding, achieving competitive results without manual data collection.

Contribution

It introduces a method to generate large-scale synthetic depth data for scene understanding, reducing reliance on manual labeling and setting new benchmarks.

Findings

Comparable performance to state-of-the-art RGBD systems on NYUv2

Sets a new benchmark on depth-based segmentation on SUN RGB-D

Shows potential for synthetic data to replace real data in training

Abstract

Scene understanding is a prerequisite to many high level tasks for any automated intelligent machine operating in real world environments. Recent attempts with supervised learning have shown promise in this direction but also highlighted the need for enormous quantity of supervised data --- performance increases in proportion to the amount of data used. However, this quickly becomes prohibitive when considering the manual labour needed to collect such data. In this work, we focus our attention on depth based semantic per-pixel labelling as a scene understanding problem and show the potential of computer graphics to generate virtually unlimited labelled data from synthetic 3D scenes. By carefully synthesizing training data with appropriate noise models we show comparable performance to state-of-the-art RGBD systems on NYUv2 dataset despite using only depth data as input and set a benchmark on depth-based segmentation on SUN RGB-D dataset. Additionally, we offer a route to generating synthesized frame or video data, and understanding of different factors influencing performance gains.