Predicate correlation learning for scene graph generation

TL;DR

This paper introduces Predicate Correlation Learning (PCL) for scene graph generation, addressing predicate overlap and long-tailed data issues by modeling predicate correlations to improve tail class performance.

Contribution

The paper proposes a novel Predicate Correlation Matrix and loss function to effectively model predicate relationships and mitigate long-tailed bias in scene graph generation.

Findings

Significant improvement in tail class performance on Visual Genome.

Effective reduction of long-tailed bias in predicate classification.

Enhanced modeling of predicate semantic overlap.

Abstract

For a typical Scene Graph Generation (SGG) method, there is often a large gap in the performance of the predicates' head classes and tail classes. This phenomenon is mainly caused by the semantic overlap between different predicates as well as the long-tailed data distribution. In this paper, a Predicate Correlation Learning (PCL) method for SGG is proposed to address the above two problems by taking the correlation between predicates into consideration. To describe the semantic overlap between strong-correlated predicate classes, a Predicate Correlation Matrix (PCM) is defined to quantify the relationship between predicate pairs, which is dynamically updated to remove the matrix's long-tailed bias. In addition, PCM is integrated into a Predicate Correlation Loss function ($L_{PC}$) to reduce discouraging gradients of unannotated classes. The proposed method is evaluated on Visual Genome benchmark, where the performance of the tail classes is significantly improved when built on the existing methods.