Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detection

TL;DR

This paper introduces a novel Part Semantic Network with a Conditional Part Attention mechanism and an Occluded Part Extrapolation strategy to improve human-object interaction detection, especially under occlusion, without relying on external annotations.

Contribution

The paper proposes a new Part Semantic Network with CPA and OPE strategies, enabling better interaction recognition and occlusion handling without external data.

Findings

Outperforms prior methods on V-COCO and HICO-DET datasets

Effectively recognizes subtle human-object interactions

Handles occlusion scenarios with extrapolation

Abstract

Human-Object Interaction Detection is a crucial aspect of human-centric scene understanding, with important applications in various domains. Despite recent progress in this field, recognizing subtle and detailed interactions remains challenging. Existing methods try to use human-related clues to alleviate the difficulty, but rely heavily on external annotations or knowledge, limiting their practical applicability in real-world scenarios. In this work, we propose a novel Part Semantic Network (PSN) to solve this problem. The core of PSN is a Conditional Part Attention (CPA) mechanism, where human features are taken as keys and values, and the object feature is used as query for the computation in a cross-attention mechanism. In this way, our model learns to automatically focus on the most informative human parts conditioned on the involved object, generating more semantically meaningful features for interaction recognition. Additionally, we propose an Occluded Part Extrapolation (OPE) strategy to facilitate interaction recognition under occluded scenarios, which teaches the model to extrapolate detailed features from partially occluded ones. Our method consistently outperforms prior approaches on the V-COCO and HICO-DET datasets, without external data or extra annotations. Additional ablation studies validate the effectiveness of each component of our proposed method.