3D-Aware Indoor Scene Synthesis with Depth Priors

TL;DR

This paper introduces a depth-guided 3D-aware indoor scene synthesis method using a dual-path generator and switchable discriminator, significantly improving the quality and 3D consistency of generated scenes from 2D data.

Contribution

The work proposes a novel depth prior integration with a dual-path generator and switchable discriminator for improved indoor scene synthesis.

Findings

Outperforms state-of-the-art methods in scene quality

Achieves high 3D consistency in generated scenes

Effectively models diverse indoor layouts

Abstract

Despite the recent advancement of Generative Adversarial Networks (GANs) in learning 3D-aware image synthesis from 2D data, existing methods fail to model indoor scenes due to the large diversity of room layouts and the objects inside. We argue that indoor scenes do not have a shared intrinsic structure, and hence only using 2D images cannot adequately guide the model with the 3D geometry. In this work, we fill in this gap by introducing depth as a 3D prior. Compared with other 3D data formats, depth better fits the convolution-based generation mechanism and is more easily accessible in practice. Specifically, we propose a dual-path generator, where one path is responsible for depth generation, whose intermediate features are injected into the other path as the condition for appearance rendering. Such a design eases the 3D-aware synthesis with explicit geometry information. Meanwhile, we introduce a switchable discriminator both to differentiate real v.s. fake domains and to predict the depth from a given input. In this way, the discriminator can take the spatial arrangement into account and advise the generator to learn an appropriate depth condition. Extensive experimental results suggest that our approach is capable of synthesizing indoor scenes with impressively good quality and 3D consistency, significantly outperforming state-of-the-art alternatives.