Feel the Force: Contact-Driven Learning from Humans

TL;DR

FeelTheForce is a robot learning system that models human tactile behavior to enable precise force-sensitive manipulation through visual and tactile data, achieving high success in real-world tasks.

Contribution

This work introduces a novel approach combining tactile sensing and vision-based modeling to learn force-aware manipulation directly from human demonstrations.

Findings

Achieved 77% success rate across 5 manipulation tasks.

Successfully transferred human tactile behavior to a robot system.

Enabled precise force control in real-world manipulation scenarios.

Abstract

Controlling fine-grained forces during manipulation remains a core challenge in robotics. While robot policies learned from robot-collected data or simulation show promise, they struggle to generalize across the diverse range of real-world interactions. Learning directly from humans offers a scalable solution, enabling demonstrators to perform skills in their natural embodiment and in everyday environments. However, visual demonstrations alone lack the information needed to infer precise contact forces. We present FeelTheForce (FTF): a robot learning system that models human tactile behavior to learn force-sensitive manipulation. Using a tactile glove to measure contact forces and a vision-based model to estimate hand pose, we train a closed-loop policy that continuously predicts the forces needed for manipulation. This policy is re-targeted to a Franka Panda robot with tactile gripper sensors using shared visual and action representations. At execution, a PD controller modulates gripper closure to track predicted forces-enabling precise, force-aware control. Our approach grounds robust low-level force control in scalable human supervision, achieving a 77% success rate across 5 force-sensitive manipulation tasks. Code and videos are available at https://feel-the-force-ftf.github.io.