ChildPlay-Hand: A Dataset of Hand Manipulations in the Wild

TL;DR

ChildPlay-Hand is a new in-the-wild third-person dataset capturing hand-object interactions with detailed annotations, including gaze, to advance HOI understanding in natural settings for tasks like object detection and manipulation stage recognition.

Contribution

The paper introduces ChildPlay-Hand, a novel dataset with per-hand annotations, gaze labels, and natural interactions, filling a gap in third-person HOI datasets for in-the-wild scenarios.

Findings

ChildPlay-Hand is a challenging benchmark for HOI modeling.

Hand-region information improves manipulation detection.

RGB and pose modalities offer complementary insights.

Abstract

Hand-Object Interaction (HOI) is gaining significant attention, particularly with the creation of numerous egocentric datasets driven by AR/VR applications. However, third-person view HOI has received less attention, especially in terms of datasets. Most third-person view datasets are curated for action recognition tasks and feature pre-segmented clips of high-level daily activities, leaving a gap for in-the-wild datasets. To address this gap, we propose ChildPlay-Hand, a novel dataset that includes person and object bounding boxes, as well as manipulation actions. ChildPlay-Hand is unique in: (1) providing per-hand annotations; (2) featuring videos in uncontrolled settings with natural interactions, involving both adults and children; (3) including gaze labels from the ChildPlay-Gaze dataset for joint modeling of manipulations and gaze. The manipulation actions cover the main stages of an HOI cycle, such as grasping, holding or operating, and different types of releasing. To illustrate the interest of the dataset, we study two tasks: object in hand detection (OiH), i.e. if a person has an object in their hand, and manipulation stages (ManiS), which is more fine-grained and targets the main stages of manipulation. We benchmark various spatio-temporal and segmentation networks, exploring body vs. hand-region information and comparing pose and RGB modalities. Our findings suggest that ChildPlay-Hand is a challenging new benchmark for modeling HOI in the wild.