Scene Graph Expansion for Semantics-Guided Image Outpainting

TL;DR

This paper introduces a novel scene graph transformer network for semantics-guided image outpainting, enabling the completion of images by understanding and expanding scene semantics at the graph level.

Contribution

The paper proposes a unique scene graph transformer that models structural information with attention at node and edge levels for improved image outpainting.

Findings

SGT effectively expands scene graphs for image completion.

Results outperform existing layout-to-image methods.

Demonstrated on MS-COCO and Visual Genome datasets.

Abstract

In this paper, we address the task of semantics-guided image outpainting, which is to complete an image by generating semantically practical content. Different from most existing image outpainting works, we approach the above task by understanding and completing image semantics at the scene graph level. In particular, we propose a novel network of Scene Graph Transformer (SGT), which is designed to take node and edge features as inputs for modeling the associated structural information. To better understand and process graph-based inputs, our SGT uniquely performs feature attention at both node and edge levels. While the former views edges as relationship regularization, the latter observes the co-occurrence of nodes for guiding the attention process. We demonstrate that, given a partial input image with its layout and scene graph, our SGT can be applied for scene graph expansion and its conversion to a complete layout. Following state-of-the-art layout-to-image conversions works, the task of image outpainting can be completed with sufficient and practical semantics introduced. Extensive experiments are conducted on the datasets of MS-COCO and Visual Genome, which quantitatively and qualitatively confirm the effectiveness of our proposed SGT and outpainting frameworks.