Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images

TL;DR

This paper introduces a novel volumetric GAN-based method for generating detailed 3D indoor scenes from semantic-segmented depth images, reducing data collection effort and improving scene quality.

Contribution

It models indoor scenes as 3D semantic volumes learned from 2.5D observations using a differentiable projection and a multi-view discriminator, enabling efficient scene generation.

Findings

Produces more accurate object shapes and layouts

Reduces workload for 3D scene modeling

Effective on various indoor datasets

Abstract

We present a method for creating 3D indoor scenes with a generative model learned from a collection of semantic-segmented depth images captured from different unknown scenes. Given a room with a specified size, our method automatically generates 3D objects in a room from a randomly sampled latent code. Different from existing methods that represent an indoor scene with the type, location, and other properties of objects in the room and learn the scene layout from a collection of complete 3D indoor scenes, our method models each indoor scene as a 3D semantic scene volume and learns a volumetric generative adversarial network (GAN) from a collection of 2.5D partial observations of 3D scenes. To this end, we apply a differentiable projection layer to project the generated 3D semantic scene volumes into semantic-segmented depth images and design a new multiple-view discriminator for learning the complete 3D scene volume from 2.5D semantic-segmented depth images. Compared to existing methods, our method not only efficiently reduces the workload of modeling and acquiring 3D scenes for training, but also produces better object shapes and their detailed layouts in the scene. We evaluate our method with different indoor scene datasets and demonstrate the advantages of our method. We also extend our method for generating 3D indoor scenes from semantic-segmented depth images inferred from RGB images of real scenes.