End-to-End Optimization of Scene Layout

TL;DR

This paper introduces an end-to-end variational model for scene layout synthesis conditioned on scene graphs, enabling flexible, diverse, and refined scene generation from various inputs.

Contribution

It presents a novel conditional scene layout generator with a differentiable rendering module for layout refinement, improving control, diversity, and accuracy over prior methods.

Findings

Higher accuracy in conditional scene synthesis

Enhanced diversity of generated layouts

Effective refinement using 2D projections

Abstract

We propose an end-to-end variational generative model for scene layout synthesis conditioned on scene graphs. Unlike unconditional scene layout generation, we use scene graphs as an abstract but general representation to guide the synthesis of diverse scene layouts that satisfy relationships included in the scene graph. This gives rise to more flexible control over the synthesis process, allowing various forms of inputs such as scene layouts extracted from sentences or inferred from a single color image. Using our conditional layout synthesizer, we can generate various layouts that share the same structure of the input example. In addition to this conditional generation design, we also integrate a differentiable rendering module that enables layout refinement using only 2D projections of the scene. Given a depth and a semantics map, the differentiable rendering module enables optimizing over the synthesized layout to fit the given input in an analysis-by-synthesis fashion. Experiments suggest that our model achieves higher accuracy and diversity in conditional scene synthesis and allows exemplar-based scene generation from various input forms.