Funnel-HOI: Top-Down Perception for Zero-Shot HOI Detection

TL;DR

Funnel-HOI introduces a top-down encoder-focused approach with a novel co-attention mechanism for improved zero-shot human-object interaction detection, achieving state-of-the-art results on benchmark datasets.

Contribution

The paper proposes a new top-down framework with an asymmetric co-attention mechanism and a novel loss for better scene understanding in HOID, especially in zero-shot scenarios.

Findings

Achieves up to 12.4% and 8.4% improvements on unseen and rare HOI categories.

Outperforms existing methods on HICO-DET and V-COCO datasets.

Effective in both fully-supervised and zero-shot settings.

Abstract

Human-object interaction detection (HOID) refers to localizing interactive human-object pairs in images and identifying the interactions. Since there could be an exponential number of object-action combinations, labeled data is limited - leading to a long-tail distribution problem. Recently, zero-shot learning emerged as a solution, with end-to-end transformer-based object detectors adapted for HOID becoming successful frameworks. However, their primary focus is designing improved decoders for learning entangled or disentangled interpretations of interactions. We advocate that HOI-specific cues must be anticipated at the encoder stage itself to obtain a stronger scene interpretation. Consequently, we build a top-down framework named Funnel-HOI inspired by the human tendency to grasp well-defined concepts first and then associate them with abstract concepts during scene understanding. We first probe an image for the presence of objects (well-defined concepts) and then probe for actions (abstract concepts) associated with them. A novel asymmetric co-attention mechanism mines these cues utilizing multimodal information (incorporating zero-shot capabilities) and yields stronger interaction representations at the encoder level. Furthermore, a novel loss is devised that considers objectaction relatedness and regulates misclassification penalty better than existing loss functions for guiding the interaction classifier. Extensive experiments on the HICO-DET and V-COCO datasets across fully-supervised and six zero-shot settings reveal our state-of-the-art performance, with up to 12.4% and 8.4% gains for unseen and rare HOI categories, respectively.