Self-Supervised Relation Alignment for Scene Graph Generation

TL;DR

This paper introduces a self-supervised relational alignment method for scene graph generation that enhances existing models by aligning masked relation predictions with supervised ones through distillation.

Contribution

It proposes a novel self-supervised regularization technique that can be integrated with any scene graph generation framework to improve performance.

Findings

Significant performance improvements on benchmark datasets.

Effective when combined with SGTR and Neural Motifs architectures.

Enhances relation prediction accuracy through relational alignment.

Abstract

The goal of scene graph generation is to predict a graph from an input image, where nodes correspond to identified and localized objects and edges to their corresponding interaction predicates. Existing methods are trained in a fully supervised manner and focus on message passing mechanisms, loss functions, and/or bias mitigation. In this work we introduce a simple-yet-effective self-supervised relational alignment regularization designed to improve the scene graph generation performance. The proposed alignment is general and can be combined with any existing scene graph generation framework, where it is trained alongside the original model's objective. The alignment is achieved through distillation, where an auxiliary relation prediction branch, that mirrors and shares parameters with the supervised counterpart, is designed. In the auxiliary branch, relational input features are partially masked prior to message passing and predicate prediction. The predictions for masked relations are then aligned with the supervised counterparts after the message passing. We illustrate the effectiveness of this self-supervised relational alignment in conjunction with two scene graph generation architectures, SGTR and Neural Motifs, and show that in both cases we achieve significantly improved performance.