Hyper-relationship Learning Network for Scene Graph Generation

TL;DR

This paper introduces HLN, a hyper-relationship learning network that enhances scene graph generation by integrating object and relationship interactions with transitive reasoning, significantly improving performance on the Visual Genome dataset.

Contribution

The paper proposes a novel HLN model utilizing hypergraphs and dual graph attention networks to perform transitive inference and better integrate graph component interactions in scene graph generation.

Findings

HLN improves recall per relationship from 11.3% to 13.1%.

HLN maintains recall per image from 19.8% to 34.9%.

HLN outperforms recent state-of-the-art methods on Visual Genome.

Abstract

Generating informative scene graphs from images requires integrating and reasoning from various graph components, i.e., objects and relationships. However, current scene graph generation (SGG) methods, including the unbiased SGG methods, still struggle to predict informative relationships due to the lack of 1) high-level inference such as transitive inference between relationships and 2) efficient mechanisms that can incorporate all interactions of graph components. To address the issues mentioned above, we devise a hyper-relationship learning network, termed HLN, for SGG. Specifically, the proposed HLN stems from hypergraphs and two graph attention networks (GATs) are designed to infer relationships: 1) the object-relationship GAT or OR-GAT to explore interactions between objects and relationships, and 2) the hyper-relationship GAT or HR-GAT to integrate transitive inference of hyper-relationships, i.e., the sequential relationships between three objects for transitive reasoning. As a result, HLN significantly improves the performance of scene graph generation by integrating and reasoning from object interactions, relationship interactions, and transitive inference of hyper-relationships. We evaluate HLN on the most popular SGG dataset, i.e., the Visual Genome dataset, and the experimental results demonstrate its great superiority over recent state-of-the-art methods. For example, the proposed HLN improves the recall per relationship from 11.3\% to 13.1\%, and maintains the recall per image from 19.8\% to 34.9\%. We will release the source code and pretrained models on GitHub.