Enhanced Data Transfer Cooperating with Artificial Triplets for Scene Graph Generation

TL;DR

This paper introduces two novel modules, FSTA and Soft Transfer, to enhance training datasets for Scene Graph Generation, significantly improving the model's ability to predict informative relational triplets.

Contribution

The work presents innovative dataset augmentation techniques, FSTA and Soft Transfer, that improve scene graph generation by focusing on challenging triplets and providing better supervision.

Findings

Achieves highest mean Recall and mean Recall on Visual Genome dataset.

Effectively augments training data with artificial triplets and soft predicate labels.

Improves model performance on informative relational triplets.

Abstract

This work focuses on training dataset enhancement of informative relational triplets for Scene Graph Generation (SGG). Due to the lack of effective supervision, the current SGG model predictions perform poorly for informative relational triplets with inadequate training samples. Therefore, we propose two novel training dataset enhancement modules: Feature Space Triplet Augmentation (FSTA) and Soft Transfer. FSTA leverages a feature generator trained to generate representations of an object in relational triplets. The biased prediction based sampling in FSTA efficiently augments artificial triplets focusing on the challenging ones. In addition, we introduce Soft Transfer, which assigns soft predicate labels to general relational triplets to make more supervisions for informative predicate classes effectively. Experimental results show that integrating FSTA and Soft Transfer achieve high levels of both Recall and mean Recall in Visual Genome dataset. The mean of Recall and mean Recall is the highest among all the existing model-agnostic methods.