PEAR: Phrase-Based Hand-Object Interaction Anticipation

TL;DR

PEAR is a novel model that jointly predicts hand-object interaction intentions and manipulations, addressing previous limitations by integrating intention and manipulation constraints, and is validated on a new dataset with superior results.

Contribution

The paper introduces PEAR, a model that jointly anticipates interaction intention and manipulation, and presents the new EGO-HOIP dataset for comprehensive evaluation.

Findings

PEAR outperforms existing methods in interaction anticipation tasks.

Cross-alignment reduces intention uncertainty effectively.

Bidirectional constraints ensure consistency between intention and manipulation.

Abstract

First-person hand-object interaction anticipation aims to predict the interaction process over a forthcoming period based on current scenes and prompts. This capability is crucial for embodied intelligence and human-robot collaboration. The complete interaction process involves both pre-contact interaction intention (i.e., hand motion trends and interaction hotspots) and post-contact interaction manipulation (i.e., manipulation trajectories and hand poses with contact). Existing research typically anticipates only interaction intention while neglecting manipulation, resulting in incomplete predictions and an increased likelihood of intention errors due to the lack of manipulation constraints. To address this, we propose a novel model, PEAR (Phrase-Based Hand-Object Interaction Anticipation), which jointly anticipates interaction intention and manipulation. To handle uncertainties in the interaction process, we employ a twofold approach. Firstly, we perform cross-alignment of verbs, nouns, and images to reduce the diversity of hand movement patterns and object functional attributes, thereby mitigating intention uncertainty. Secondly, we establish bidirectional constraints between intention and manipulation using dynamic integration and residual connections, ensuring consistency among elements and thus overcoming manipulation uncertainty. To rigorously evaluate the performance of the proposed model, we collect a new task-relevant dataset, EGO-HOIP, with comprehensive annotations. Extensive experimental results demonstrate the superiority of our method.