Learning Human-Object Interaction as Groups

TL;DR

This paper introduces GroupHOI, a novel framework for human-object interaction detection that models interactions from a group perspective by leveraging geometric proximity and semantic similarity, improving performance on multiple benchmarks.

Contribution

The paper proposes a group-based relation modeling approach for HOI detection, incorporating learnable proximity clustering and enhanced transformer decoders for better context aggregation.

Findings

Outperforms state-of-the-art methods on HICO-DET and V-COCO benchmarks.

Achieves leading results on the challenging NVI-DET task.

Effectively models higher-order interactions within groups.

Abstract

Human-Object Interaction Detection (HOI-DET) aims to localize human-object pairs and identify their interactive relationships. To aggregate contextual cues, existing methods typically propagate information across all detected entities via self-attention mechanisms, or establish message passing between humans and objects with bipartite graphs. However, they primarily focus on pairwise relationships, overlooking that interactions in real-world scenarios often emerge from collective behaviors (multiple humans and objects engaging in joint activities). In light of this, we revisit relation modeling from a group view and propose GroupHOI, a framework that propagates contextual information in terms of geometric proximity and semantic similarity. To exploit the geometric proximity, humans and objects are grouped into distinct clusters using a learnable proximity estimator based on spatial features derived from bounding boxes. In each group, a soft correspondence is computed via self-attention to aggregate and dispatch contextual cues. To incorporate the semantic similarity, we enhance the vanilla transformer-based interaction decoder with local contextual cues from HO-pair features. Extensive experiments on HICO-DET and V-COCO benchmarks demonstrate the superiority of GroupHOI over the state-of-the-art methods. It also exhibits leading performance on the more challenging Nonverbal Interaction Detection (NVI-DET) task, which involves varied forms of higher-order interactions within groups.