Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation

TL;DR

This paper introduces a novel training framework for scene graph generation that leverages relation informativeness to improve model reasoning, generalization, and zero-shot performance on the Visual Genome benchmark.

Contribution

It proposes a model-agnostic method that imputes informative relation labels for training, enhancing existing SGG models without altering their architecture.

Findings

Significant performance improvements on standard metrics.

Enhanced zero-shot triplet prediction capabilities.

Compatibility with multiple state-of-the-art SGG methods.

Abstract

Scene graph generation (SGG) aims to capture a wide variety of interactions between pairs of objects, which is essential for full scene understanding. Existing SGG methods trained on the entire set of relations fail to acquire complex reasoning about visual and textual correlations due to various biases in training data. Learning on trivial relations that indicate generic spatial configuration like 'on' instead of informative relations such as 'parked on' does not enforce this complex reasoning, harming generalization. To address this problem, we propose a novel framework for SGG training that exploits relation labels based on their informativeness. Our model-agnostic training procedure imputes missing informative relations for less informative samples in the training data and trains a SGG model on the imputed labels along with existing annotations. We show that this approach can successfully be used in conjunction with state-of-the-art SGG methods and improves their performance significantly in multiple metrics on the standard Visual Genome benchmark. Furthermore, we obtain considerable improvements for unseen triplets in a more challenging zero-shot setting.