Transforming Image Generation from Scene Graphs

TL;DR

This paper introduces a transformer-based model conditioned on scene graphs for controllable image generation, enabling iterative modification and better semantic constraint satisfaction, demonstrated on CIFAR10 and MNIST datasets.

Contribution

It presents a novel autoregressive transformer architecture with a decoder for scene graph conditioned image synthesis, enhancing control and flexibility over the generation process.

Findings

Model effectively satisfies semantic constraints from scene graphs.

It models relations between visual objects considering user input.

Demonstrates promising results on CIFAR10 and MNIST datasets.

Abstract

Generating images from semantic visual knowledge is a challenging task, that can be useful to condition the synthesis process in complex, subtle, and unambiguous ways, compared to alternatives such as class labels or text descriptions. Although generative methods conditioned by semantic representations exist, they do not provide a way to control the generation process aside from the specification of constraints between objects. As an example, the possibility to iteratively generate or modify images by manually adding specific items is a desired property that, to our knowledge, has not been fully investigated in the literature. In this work we propose a transformer-based approach conditioned by scene graphs that, conversely to recent transformer-based methods, also employs a decoder to autoregressively compose images, making the synthesis process more effective and controllable. The proposed architecture is composed by three modules: 1) a graph convolutional network, to encode the relationships of the input graph; 2) an encoder-decoder transformer, which autoregressively composes the output image; 3) an auto-encoder, employed to generate representations used as input/output of each generation step by the transformer. Results obtained on CIFAR10 and MNIST images show that our model is able to satisfy semantic constraints defined by a scene graph and to model relations between visual objects in the scene by taking into account a user-provided partial rendering of the desired target.