Multimodal Adversarial Quality Policy for Safe Grasping

TL;DR

This paper introduces MAQP, a multimodal adversarial policy that enhances the safety of vision-guided robot grasping by effectively managing RGB and depth data through novel optimization and balancing strategies.

Contribution

The work presents a novel multimodal adversarial framework with dual-patch optimization and gradient balancing strategies for safer robot grasping in RGBD environments.

Findings

MAQP improves safety in vision-guided grasping tasks.

Experimental results show enhanced robustness against adversarial attacks.

Framework achieves superior performance on benchmark datasets and real cobots.

Abstract

Vision-guided robot grasping based on Deep Neural Networks (DNNs) generalizes well but poses safety risks in the Human-Robot Interaction (HRI). Recent works solved it by designing benign adversarial attacks and patches with RGB modality, yet depth-independent characteristics limit their effectiveness on RGBD modality. In this work, we propose the Multimodal Adversarial Quality Policy (MAQP) to realize multimodal safe grasping. Our framework introduces two key components. First, the Heterogeneous Dual-Patch Optimization Scheme (HDPOS) mitigates the distribution discrepancy between RGB and depth modalities in patch generation by adopting modality-specific initialization strategies, employing a Gaussian distribution for depth patches and a uniform distribution for RGB patches, while jointly optimizing both modalities under a unified objective function. Second, the Gradient-Level Modality Balancing Strategy (GLMBS) is designed to resolve the optimization imbalance from RGB and Depth patches in patch shape adaptation by reweighting gradient contributions based on per-channel sensitivity analysis and applying distance-adaptive perturbation bounds. We conduct extensive experiments on the benchmark datasets and a cobot, showing the effectiveness of MAQP.