FBI: Learning Dexterous In-hand Manipulation with Dynamic Visuotactile Shortcut Policy

TL;DR

This paper presents FBI, a novel visuotactile imitation learning framework that dynamically fuses tactile and visual data for dexterous in-hand manipulation, outperforming previous static fusion methods in simulation and real-world tasks.

Contribution

FBI introduces a dynamics-aware latent model and transformer-based fusion for real-time visuotactile manipulation, advancing beyond prior static fusion approaches.

Findings

FBI outperforms baseline methods in simulation and real-world tasks.

The method effectively fuses tactile and visual data for manipulation.

Code and models are publicly available.

Abstract

Dexterous in-hand manipulation is a long-standing challenge in robotics due to complex contact dynamics and partial observability. While humans synergize vision and touch for such tasks, robotic approaches often prioritize one modality, therefore limiting adaptability. This paper introduces Flow Before Imitation (FBI), a visuotactile imitation learning framework that dynamically fuses tactile interactions with visual observations through motion dynamics. Unlike prior static fusion methods, FBI establishes a causal link between tactile signals and object motion via a dynamics-aware latent model. FBI employs a transformer-based interaction module to fuse flow-derived tactile features with visual inputs, training a one-step diffusion policy for real-time execution. Extensive experiments demonstrate that the proposed method outperforms the baseline methods in both simulation and the real world on two customized in-hand manipulation tasks and three standard dexterous manipulation tasks. Code, models, and more results are available in the website https://sites.google.com/view/dex-fbi.