OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction

TL;DR

OpenTouch is a pioneering in-the-wild egocentric dataset linking first-person video with full-hand tactile data, enabling improved perception and robotic manipulation through multimodal learning.

Contribution

The paper introduces the first comprehensive in-the-wild egocentric tactile dataset with synchronized video, touch, and pose data, along with benchmarks for perception and action grounding.

Findings

Tactile signals enhance grasp understanding.

Cross-modal alignment is strengthened by tactile data.

Reliable retrieval of tactile cues from in-the-wild video queries.

Abstract

The human hand is our primary interface to the physical world, yet egocentric perception rarely knows when, where, or how forcefully it makes contact. Robust wearable tactile sensors are scarce, and no existing in-the-wild datasets align first-person video with full-hand touch. To bridge the gap between visual perception and physical interaction, we present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset, containing 5.1 hours of synchronized video-touch-pose data and 2,900 curated clips with detailed text annotations. Using OpenTouch, we introduce retrieval and classification benchmarks that probe how touch grounds perception and action. We show that tactile signals provide a compact yet powerful cue for grasp understanding, strengthen cross-modal alignment, and can be reliably retrieved from in-the-wild video queries. By releasing this annotated vision-touch-pose dataset and benchmark, we aim to advance multimodal egocentric perception, embodied learning, and contact-rich robotic manipulation.