TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

TL;DR

This paper introduces EgoTouch, a large-scale egocentric dataset with tactile supervision, and TouchAnything, a framework for predicting tactile information from visual data to enhance embodied interaction models.

Contribution

The paper presents EgoTouch dataset with synchronized visual and tactile data, and a vision-to-touch prediction framework that leverages multi-view inputs for improved tactile inference.

Findings

Incorporating wrist-mounted views improves tactile prediction accuracy.

TouchAnything achieves up to 5.0% improvement in Contact IoU.

The dataset and code will be publicly released.

Abstract

Egocentric human video data, which captures rich human-environment interactions and can be collected at scale, has become a key driver of embodied intelligence research. However, existing egocentric datasets typically lack tactile sensing, a critical modality that provides direct cues about contact, force, and pressure in human-object interaction. Without such signals, models struggle to learn physically grounded representations of real-world interaction dynamics. While tactile sensors provide these cues, deploying high-quality tactile hardware at scale remains expensive and cumbersome. This raises a central question: can tactile feedback be inferred directly from visual observations, enabling scalable tactile supervision for egocentric video data and supporting physically grounded embodied learning? To enable research in this direction, we introduce EgoTouch, a large-scale multi-view egocentric dataset with dense tactile supervision for bimanual hand-object interaction. EgoTouch comprises 208 manipulation tasks spanning 1,891 episodes in diverse indoor and outdoor environments, with synchronized multi-view RGB (head-mounted egocentric and dual wrist-mounted cameras), bimanual 3D hand pose, and continuous pressure maps from wearable tactile sensors. Building on EgoTouch, we introduce TouchAnything, a baseline multi-view vision-to-touch prediction framework that uses the egocentric view as the primary input and flexibly leverages available wrist-mounted views at inference time. Experiments show that incorporating wrist-mounted views generally improves tactile prediction over egocentric-only input, achieving up to 5.0% relative improvement in Contact IoU and 6.1% relative improvement in Volumetric IoU. We will publicly release the dataset, code, and benchmark.