Visual-Semantic Graph Attention Networks for Human-Object Interaction Detection

TL;DR

This paper introduces a dual-graph attention network that enhances human-object interaction detection by dynamically aggregating visual, spatial, and semantic context, leading to improved disambiguation and performance.

Contribution

It proposes a novel dual-graph attention architecture that leverages subsidiary relations and multi-modal cues for better HOI detection.

Findings

Achieves comparable results on V-COCO and HICO-DET benchmarks.

Effectively disambiguates interactions using subsidiary relations.

Demonstrates the importance of contextual information in HOI detection.

Abstract

In scene understanding, robotics benefit from not only detecting individual scene instances but also from learning their possible interactions. Human-Object Interaction (HOI) Detection infers the action predicate on a <human, predicate, object> triplet. Contextual information has been found critical in inferring interactions. However, most works only use local features from single human-object pair for inference. Few works have studied the disambiguating contribution of subsidiary relations made available via graph networks. Similarly, few have learned to effectively leverage visual cues along with the intrinsic semantic regularities contained in HOIs. We contribute a dual-graph attention network that effectively aggregates contextual visual, spatial, and semantic information dynamically from primary human-object relations as well as subsidiary relations through attention mechanisms for strong disambiguating power. We achieve comparable results on two benchmarks: V-COCO and HICO-DET. Code is available at \url{https://github.com/birlrobotics/vs-gats}.