Exploiting Scene Graphs for Human-Object Interaction Detection

TL;DR

This paper introduces SG2HOI, a novel approach that leverages scene graphs to incorporate high-level semantic relationships, significantly improving human-object interaction detection accuracy on benchmark datasets.

Contribution

The paper proposes a new method that embeds scene graphs into HOI detection, utilizing relation-aware message passing to enhance contextual understanding.

Findings

Outperforms state-of-the-art on V-COCO and HICO-DET datasets

Demonstrates the effectiveness of scene graph integration in HOI detection

Provides a new framework for relation-aware contextual modeling

Abstract

Human-Object Interaction (HOI) detection is a fundamental visual task aiming at localizing and recognizing interactions between humans and objects. Existing works focus on the visual and linguistic features of humans and objects. However, they do not capitalise on the high-level and semantic relationships present in the image, which provides crucial contextual and detailed relational knowledge for HOI inference. We propose a novel method to exploit this information, through the scene graph, for the Human-Object Interaction (SG2HOI) detection task. Our method, SG2HOI, incorporates the SG information in two ways: (1) we embed a scene graph into a global context clue, serving as the scene-specific environmental context; and (2) we build a relation-aware message-passing module to gather relationships from objects' neighborhood and transfer them into interactions. Empirical evaluation shows that our SG2HOI method outperforms the state-of-the-art methods on two benchmark HOI datasets: V-COCO and HICO-DET. Code will be available at https://github.com/ht014/SG2HOI.