Unconditional Scene Graph Generation

TL;DR

This paper introduces SceneGraphGen, a deep auto-regressive model for the unconditional generation of semantic scene graphs, capturing complex scene structures and enabling applications like image synthesis and anomaly detection.

Contribution

It presents the first model for unconditional scene graph generation using a hierarchical recurrent architecture, learning the distribution over labeled, directed graphs.

Findings

Generated scene graphs are diverse and semantically realistic.

The model can be applied to image synthesis, anomaly detection, and scene graph completion.

Abstract

Despite recent advancements in single-domain or single-object image generation, it is still challenging to generate complex scenes containing diverse, multiple objects and their interactions. Scene graphs, composed of nodes as objects and directed-edges as relationships among objects, offer an alternative representation of a scene that is more semantically grounded than images. We hypothesize that a generative model for scene graphs might be able to learn the underlying semantic structure of real-world scenes more effectively than images, and hence, generate realistic novel scenes in the form of scene graphs. In this work, we explore a new task for the unconditional generation of semantic scene graphs. We develop a deep auto-regressive model called SceneGraphGen which can directly learn the probability distribution over labelled and directed graphs using a hierarchical recurrent architecture. The model takes a seed object as input and generates a scene graph in a sequence of steps, each step generating an object node, followed by a sequence of relationship edges connecting to the previous nodes. We show that the scene graphs generated by SceneGraphGen are diverse and follow the semantic patterns of real-world scenes. Additionally, we demonstrate the application of the generated graphs in image synthesis, anomaly detection and scene graph completion.