Generative Compositional Augmentations for Scene Graph Prediction

TL;DR

This paper introduces a method to generate synthetic scene graphs using GANs to improve the ability of models to recognize rare or unseen object relationships, enhancing compositional generalization in scene graph prediction.

Contribution

It proposes a novel approach to synthesize plausible scene graphs for rare compositions, addressing the challenge of compositional generalization in scene graph prediction.

Findings

Marginal improvements in zero-shot and few-shot metrics on Visual Genome

Effective synthesis of rare scene graph compositions

Analysis of limitations and future directions

Abstract

Inferring objects and their relationships from an image in the form of a scene graph is useful in many applications at the intersection of vision and language. We consider a challenging problem of compositional generalization that emerges in this task due to a long tail data distribution. Current scene graph generation models are trained on a tiny fraction of the distribution corresponding to the most frequent compositions, e.g. <cup, on, table>. However, test images might contain zero- and few-shot compositions of objects and relationships, e.g. <cup, on, surfboard>. Despite each of the object categories and the predicate (e.g. 'on') being frequent in the training data, the models often fail to properly understand such unseen or rare compositions. To improve generalization, it is natural to attempt increasing the diversity of the training distribution. However, in the graph domain this is non-trivial. To that end, we propose a method to synthesize rare yet plausible scene graphs by perturbing real ones. We then propose and empirically study a model based on conditional generative adversarial networks (GANs) that allows us to generate visual features of perturbed scene graphs and learn from them in a joint fashion. When evaluated on the Visual Genome dataset, our approach yields marginal, but consistent improvements in zero- and few-shot metrics. We analyze the limitations of our approach indicating promising directions for future research.