Grasp Stability Assessment Through Attention-Guided Cross-Modality Fusion and Transfer Learning

TL;DR

This paper introduces an attention-guided cross-modality fusion framework combining visual and tactile data for grasp stability assessment, utilizing simulation data and transfer learning to improve real-world applicability.

Contribution

It proposes a novel fusion architecture with attention mechanisms and a transfer learning strategy to enhance grasp stability prediction from multimodal data.

Findings

Achieves about 10% improvement over baseline models.

Demonstrates effective sim-to-real transfer for robotic grasping.

Establishes a large multimodal dataset via physics simulation.

Abstract

Extensive research has been conducted on assessing grasp stability, a crucial prerequisite for achieving optimal grasping strategies, including the minimum force grasping policy. However, existing works employ basic feature-level fusion techniques to combine visual and tactile modalities, resulting in the inadequate utilization of complementary information and the inability to model interactions between unimodal features. This work proposes an attention-guided cross-modality fusion architecture to comprehensively integrate visual and tactile features. This model mainly comprises convolutional neural networks (CNNs), self-attention, and cross-attention mechanisms. In addition, most existing methods collect datasets from real-world systems, which is time-consuming and high-cost, and the datasets collected are comparatively limited in size. This work establishes a robotic grasping system through physics simulation to collect a multimodal dataset. To address the sim-to-real transfer gap, we propose a migration strategy encompassing domain randomization and domain adaptation techniques. The experimental results demonstrate that the proposed fusion framework achieves markedly enhanced prediction performance (approximately 10%) compared to other baselines. Moreover, our findings suggest that the trained model can be reliably transferred to real robotic systems, indicating its potential to address real-world challenges.