Semantic Diversity-aware Prototype-based Learning for Unbiased Scene Graph Generation

TL;DR

This paper introduces a novel prototype-based learning framework that captures semantic diversity in predicates to improve unbiased scene graph generation, addressing the limitations of existing models that overlook predicate variability.

Contribution

The proposed Semantic Diversity-aware Prototype-based Learning (DPL) framework is model-agnostic and effectively models predicate semantic diversity for unbiased scene graph generation.

Findings

Significant performance improvements on existing SGG models.

Effective understanding of predicate semantic diversity.

Model-agnostic framework adaptable to various models.

Abstract

The scene graph generation (SGG) task involves detecting objects within an image and predicting predicates that represent the relationships between the objects. However, in SGG benchmark datasets, each subject-object pair is annotated with a single predicate even though a single predicate may exhibit diverse semantics (i.e., semantic diversity), existing SGG models are trained to predict the one and only predicate for each pair. This in turn results in the SGG models to overlook the semantic diversity that may exist in a predicate, thus leading to biased predictions. In this paper, we propose a novel model-agnostic Semantic Diversity-aware Prototype-based Learning (DPL) framework that enables unbiased predictions based on the understanding of the semantic diversity of predicates. Specifically, DPL learns the regions in the semantic space covered by each predicate to distinguish among the various different semantics that a single predicate can represent. Extensive experiments demonstrate that our proposed model-agnostic DPL framework brings significant performance improvement on existing SGG models, and also effectively understands the semantic diversity of predicates.