Relationship-Aware Spatial Perception Fusion for Realistic Scene Layout Generation

TL;DR

This paper introduces a novel framework that uses spatial constraints and contextual fusion to generate realistic, complex scene layouts from textual scene graphs, improving the logical placement of multiple objects.

Contribution

The paper presents a new method combining spatial constraints and contextual fusion modules to enhance scene layout generation from scene graphs.

Findings

Generated layouts are more realistic and logical.

Framework outperforms existing methods in quantitative metrics.

User studies favor the proposed approach.

Abstract

The significant progress on Generative Adversarial Networks (GANs) have made it possible to generate surprisingly realistic images for single object based on natural language descriptions. However, controlled generation of images for multiple entities with explicit interactions is still difficult to achieve due to the scene layout generation heavily suffer from the diversity object scaling and spatial locations. In this paper, we proposed a novel framework for generating realistic image layout from textual scene graphs. In our framework, a spatial constraint module is designed to fit reasonable scaling and spatial layout of object pairs with considering relationship between them. Moreover, a contextual fusion module is introduced for fusing pair-wise spatial information in terms of object dependency in scene graph. By using these two modules, our proposed framework tends to generate more commonsense layout which is helpful for realistic image generation. Experimental results including quantitative results, qualitative results and user studies on two different scene graph datasets demonstrate our proposed framework's ability to generate complex and logical layout with multiple objects from scene graph.