RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning

TL;DR

This paper introduces RA-SGG, a retrieval-augmented framework for scene graph generation that addresses long-tailed predicate distribution and semantic ambiguity by leveraging multi-label classification and multi-prototype learning, improving accuracy on benchmark datasets.

Contribution

The paper proposes a novel retrieval-augmented multi-label learning approach for scene graph generation, effectively mitigating bias and semantic ambiguity issues.

Findings

Outperforms state-of-the-art by up to 3.6% on VG and 5.9% on GQA datasets.

Effectively alleviates bias caused by long-tailed predicate distribution.

Enhances predicate prediction accuracy through multi-label augmentation.

Abstract

Scene Graph Generation (SGG) research has suffered from two fundamental challenges: the long-tailed predicate distribution and semantic ambiguity between predicates. These challenges lead to a bias towards head predicates in SGG models, favoring dominant general predicates while overlooking fine-grained predicates. In this paper, we address the challenges of SGG by framing it as multi-label classification problem with partial annotation, where relevant labels of fine-grained predicates are missing. Under the new frame, we propose Retrieval-Augmented Scene Graph Generation (RA-SGG), which identifies potential instances to be multi-labeled and enriches the single-label with multi-labels that are semantically similar to the original label by retrieving relevant samples from our established memory bank. Based on augmented relations (i.e., discovered multi-labels), we apply multi-prototype learning to train our SGG model. Several comprehensive experiments have demonstrated that RA-SGG outperforms state-of-the-art baselines by up to 3.6% on VG and 5.9% on GQA, particularly in terms of F@K, showing that RA-SGG effectively alleviates the issue of biased prediction caused by the long-tailed distribution and semantic ambiguity of predicates.