Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation

TL;DR

This paper introduces an adaptive visuo-tactile fusion method with predictive force attention, enabling robots to better utilize multi-sensory data for dexterous manipulation, achieving high success rates in real-world tasks.

Contribution

It proposes a force-guided attention fusion module and a self-supervised force prediction task to enhance multi-sensory data integration without human labeling.

Findings

Achieves 93% success rate on three real-world tasks.

Effectively adjusts attention to visual and tactile modalities during manipulation.

Improves tactile data reinforcement through self-supervised learning.

Abstract

Effectively utilizing multi-sensory data is important for robots to generalize across diverse tasks. However, the heterogeneous nature of these modalities makes fusion challenging. Existing methods propose strategies to obtain comprehensively fused features but often ignore the fact that each modality requires different levels of attention at different manipulation stages. To address this, we propose a force-guided attention fusion module that adaptively adjusts the weights of visual and tactile features without human labeling. We also introduce a self-supervised future force prediction auxiliary task to reinforce the tactile modality, improve data imbalance, and encourage proper adjustment. Our method achieves an average success rate of 93% across three fine-grained, contactrich tasks in real-world experiments. Further analysis shows that our policy appropriately adjusts attention to each modality at different manipulation stages. The videos can be viewed at https://adaptac-dex.github.io/.