Unbiased Scene Graph Generation by Type-Aware Message Passing on Heterogeneous and Dual Graphs

TL;DR

This paper introduces TA-HDG, a novel approach for unbiased scene graph generation that models complex interactions using type-aware message passing on heterogeneous and dual graphs, improving tail class prediction.

Contribution

The paper proposes a Type-Aware Message Passing method combined with Interactive Graph Construction to better model relations and reduce meaningless edges in scene graph generation.

Findings

Improves R@K and mR@K metrics on two datasets.

Effectively predicts tail classes without sacrificing head class performance.

Enhances understanding of complex object-relation interactions.

Abstract

Although great progress has been made in the research of unbiased scene graph generation, issues still hinder improving the predictive performance of both head and tail classes. An unbiased scene graph generation (TA-HDG) is proposed to address these issues. For modeling interactive and non-interactive relations, the Interactive Graph Construction is proposed to model the dependence of relations on objects by combining heterogeneous and dual graph, when modeling relations between multiple objects. It also implements a subject-object pair selection strategy to reduce meaningless edges. Moreover, the Type-Aware Message Passing enhances the understanding of complex interactions by capturing intra- and inter-type context in the Intra-Type and Inter-Type stages. The Intra-Type stage captures the semantic context of inter-relaitons and inter-objects. On this basis, the Inter-Type stage captures the context between objects and relations for interactive and non-interactive relations, respectively. Experiments on two datasets show that TA-HDG achieves improvements in the metrics of R@K and mR@K, which proves that TA-HDG can accurately predict the tail class while maintaining the competitive performance of the head class.