Prototype-based Embedding Network for Scene Graph Generation

TL;DR

This paper introduces PE-Net, a prototype-based embedding network that improves scene graph generation by modeling entities and predicates with class-wise prototypes, leading to more robust relation prediction.

Contribution

It proposes a novel prototype-based embedding approach and associated learning and regularization techniques to enhance relation recognition in scene graph generation.

Findings

Achieves state-of-the-art performance on Visual Genome dataset.

Outperforms existing methods on Open Images dataset.

Demonstrates robustness in relation prediction tasks.

Abstract

Current Scene Graph Generation (SGG) methods explore contextual information to predict relationships among entity pairs. However, due to the diverse visual appearance of numerous possible subject-object combinations, there is a large intra-class variation within each predicate category, e.g., "man-eating-pizza, giraffe-eating-leaf", and the severe inter-class similarity between different classes, e.g., "man-holding-plate, man-eating-pizza", in model's latent space. The above challenges prevent current SGG methods from acquiring robust features for reliable relation prediction. In this paper, we claim that the predicate's category-inherent semantics can serve as class-wise prototypes in the semantic space for relieving the challenges. To the end, we propose the Prototype-based Embedding Network (PE-Net), which models entities/predicates with prototype-aligned compact and distinctive representations and thereby establishes matching between entity pairs and predicates in a common embedding space for relation recognition. Moreover, Prototype-guided Learning (PL) is introduced to help PE-Net efficiently learn such entitypredicate matching, and Prototype Regularization (PR) is devised to relieve the ambiguous entity-predicate matching caused by the predicate's semantic overlap. Extensive experiments demonstrate that our method gains superior relation recognition capability on SGG, achieving new state-of-the-art performances on both Visual Genome and Open Images datasets.