Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection

TL;DR

This paper introduces a novel method to improve one-stage human-object interaction detection by extracting disentangled, interpretable interaction representations through specialized attention mechanisms and pose estimation, achieving state-of-the-art results.

Contribution

The paper proposes Shunted Cross-Attention and Interaction-aware Pose Estimation modules to extract disentangled interaction features, enhancing interpretability and performance of one-stage HOI detectors.

Findings

Achieves state-of-the-art performance on HICO-DET and V-COCO benchmarks.

Effectively disentangles human appearance, object appearance, and pose features.

Improves interpretability of interaction representations in one-stage detectors.

Abstract

Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction; however, the interaction representations obtained using this method are entangled and lack interpretability. In contrast, traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner. In this paper, we improve the performance of one-stage methods by enabling them to extract disentangled interaction representations. First, we propose Shunted Cross-Attention (SCA) to extract human appearance, object appearance, and global context features using different cross-attention heads. This is achieved by imposing different masks on the cross-attention maps produced by the different heads. Second, we introduce the Interaction-aware Pose Estimation (IPE) task to learn interaction-relevant human pose features using a disentangled decoder. This is achieved with a novel attention module that accurately captures the human keypoints relevant to the current interaction category. Finally, our approach fuses the appearance feature and pose feature via element-wise addition to form the interaction representation. Experimental results show that our approach can be readily applied to existing one-stage HOI detectors. Moreover, we achieve state-of-the-art performance on two benchmarks: HICO-DET and V-COCO.