FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation

TL;DR

FreeTacMan introduces a wearable, human-centric data collection system that captures visuo-tactile data for contact-rich robot manipulation, significantly improving data efficiency and diversity for policy learning.

Contribution

It presents a novel, robot-free, wearable visuo-tactile data collection system that enables large-scale, diverse datasets for contact-rich manipulation tasks.

Findings

Collected over 3 million visuo-tactile images and 10,000 trajectories.

Achieved improved data collection efficiency over prior methods.

Enabled effective policy learning from self-collected datasets.

Abstract

Enabling robots with contact-rich manipulation remains a pivotal challenge in robot learning, which is substantially hindered by the data collection gap, including its inefficiency and limited sensor setup. While prior work has explored handheld paradigms, their rod-based mechanical structures remain rigid and unintuitive, providing limited tactile feedback and posing challenges for operators. Motivated by the dexterity and force feedback of human motion, we propose FreeTacMan, a human-centric and robot-free data collection system for accurate and efficient robot manipulation. Concretely, we design a wearable gripper with visuo-tactile sensors for data collection, which can be worn by human fingers for intuitive control. A high-precision optical tracking system is introduced to capture end-effector poses while synchronizing visual and tactile feedback simultaneously. We leverage FreeTacMan to collect a large-scale multimodal dataset, comprising over 3000k paired visuo-tactile images with end-effector poses, 10k demonstration trajectories across 50 diverse contact-rich manipulation tasks. FreeTacMan achieves multiple improvements in data collection performance over prior works and enables effective policy learning from self-collected datasets. By open-sourcing the hardware and the dataset, we aim to facilitate reproducibility and support research in visuo-tactile manipulation.