Exploring the Hierarchy in Relation Labels for Scene Graph Generation

TL;DR

This paper introduces a hierarchy-guided approach for scene graph generation that leverages predicate hierarchies to improve feature learning, significantly boosting performance over existing methods.

Contribution

It proposes the Hierarchy Guided Feature Learning and Hierarchy Guided Module to incorporate predicate hierarchies into scene graph generation, enhancing feature representation and accuracy.

Findings

Achieved up to 33% relative gain in Recall@50

Improved performance across multiple datasets

Effective utilization of predicate hierarchies

Abstract

By assigning each relationship a single label, current approaches formulate the relationship detection as a classification problem. Under this formulation, predicate categories are treated as completely different classes. However, different from the object labels where different classes have explicit boundaries, predicates usually have overlaps in their semantic meanings. For example, sit\_on and stand\_on have common meanings in vertical relationships but different details of how these two objects are vertically placed. In order to leverage the inherent structures of the predicate categories, we propose to first build the language hierarchy and then utilize the Hierarchy Guided Feature Learning (HGFL) strategy to learn better region features of both the coarse-grained level and the fine-grained level. Besides, we also propose the Hierarchy Guided Module (HGM) to utilize the coarse-grained level to guide the learning of fine-grained level features. Experiments show that the proposed simple yet effective method can improve several state-of-the-art baselines by a large margin (up to $33\%$ relative gain) in terms of Recall@50 on the task of Scene Graph Generation in different datasets.