Exploiting Relationship for Complex-scene Image Generation

TL;DR

This paper introduces a relationship-aware framework for complex-scene image generation using scene graphs, improving layout plausibility and object coherence over prior object-driven methods.

Contribution

It proposes three key updates: scene graph-based layout inference, a relation-guided generator, and a scene graph discriminator, advancing the realism and consistency of generated complex scenes.

Findings

Outperforms prior methods in IS and FID metrics.

Produces more logical layouts and object appearances.

Effective in complex scene synthesis on Visual Genome and HICO-DET datasets.

Abstract

The significant progress on Generative Adversarial Networks (GANs) has facilitated realistic single-object image generation based on language input. However, complex-scene generation (with various interactions among multiple objects) still suffers from messy layouts and object distortions, due to diverse configurations in layouts and appearances. Prior methods are mostly object-driven and ignore their inter-relations that play a significant role in complex-scene images. This work explores relationship-aware complex-scene image generation, where multiple objects are inter-related as a scene graph. With the help of relationships, we propose three major updates in the generation framework. First, reasonable spatial layouts are inferred by jointly considering the semantics and relationships among objects. Compared to standard location regression, we show relative scales and distances serve a more reliable target. Second, since the relations between objects significantly influence an object's appearance, we design a relation-guided generator to generate objects reflecting their relationships. Third, a novel scene graph discriminator is proposed to guarantee the consistency between the generated image and the input scene graph. Our method tends to synthesize plausible layouts and objects, respecting the interplay of multiple objects in an image. Experimental results on Visual Genome and HICO-DET datasets show that our proposed method significantly outperforms prior arts in terms of IS and FID metrics. Based on our user study and visual inspection, our method is more effective in generating logical layout and appearance for complex-scenes.