GelFusion: Enhancing Robotic Manipulation under Visual Constraints via Visuotactile Fusion

TL;DR

GelFusion is a novel framework that fuses high-resolution visuotactile data with visual inputs using a cross-attention mechanism to improve robotic manipulation tasks under visual constraints.

Contribution

The paper introduces GelFusion, a visuotactile fusion framework with a dual-channel feature representation and cross-attention mechanism for enhanced policy learning in robotics.

Findings

GelFusion outperforms baselines in contact-rich tasks.

The dual-channel feature representation improves contact information richness.

Cross-attention fusion effectively integrates visuotactile data into policies.

Abstract

Visuotactile sensing offers rich contact information that can help mitigate performance bottlenecks in imitation learning, particularly under vision-limited conditions, such as ambiguous visual cues or occlusions. Effectively fusing visual and visuotactile modalities, however, presents ongoing challenges. We introduce GelFusion, a framework designed to enhance policies by integrating visuotactile feedback, specifically from high-resolution GelSight sensors. GelFusion using a vision-dominated cross-attention fusion mechanism incorporates visuotactile information into policy learning. To better provide rich contact information, the framework's core component is our dual-channel visuotactile feature representation, simultaneously leveraging both texture-geometric and dynamic interaction features. We evaluated GelFusion on three contact-rich tasks: surface wiping, peg insertion, and fragile object pick-and-place. Outperforming baselines, GelFusion shows the value of its structure in improving the success rate of policy learning.