Discovering A Variety of Objects in Spatio-Temporal Human-Object Interactions

TL;DR

This paper introduces a new benchmark for spatio-temporal human-object interaction detection with diverse objects, and proposes a Hierarchical Probe Network to improve object discovery using spatio-temporal cues, revealing current limitations.

Contribution

The paper presents a new AVA-based benchmark with 51 interactions and over 1,000 objects, and proposes a Hierarchical Probe Network for improved object discovery in videos.

Findings

HPN outperforms existing methods in object discovery tasks.

Current detectors struggle with localizing diverse/unseen objects.

The benchmark reveals limitations of current vision systems in complex interactions.

Abstract

Spatio-temporal Human-Object Interaction (ST-HOI) detection aims at detecting HOIs from videos, which is crucial for activity understanding. In daily HOIs, humans often interact with a variety of objects, e.g., holding and touching dozens of household items in cleaning. However, existing whole body-object interaction video benchmarks usually provide limited object classes. Here, we introduce a new benchmark based on AVA: Discovering Interacted Objects (DIO) including 51 interactions and 1,000+ objects. Accordingly, an ST-HOI learning task is proposed expecting vision systems to track human actors, detect interactions and simultaneously discover interacted objects. Even though today's detectors/trackers excel in object detection/tracking tasks, they perform unsatisfied to localize diverse/unseen objects in DIO. This profoundly reveals the limitation of current vision systems and poses a great challenge. Thus, how to leverage spatio-temporal cues to address object discovery is explored, and a Hierarchical Probe Network (HPN) is devised to discover interacted objects utilizing hierarchical spatio-temporal human/context cues. In extensive experiments, HPN demonstrates impressive performance. Data and code are available at https://github.com/DirtyHarryLYL/HAKE-AVA.