MagicGripper: A Multimodal Sensor-Integrated Gripper for Contact-Rich Robotic Manipulation

TL;DR

MagicGripper is a compact, multimodal sensor-integrated robotic gripper that combines tactile, proximity, and visual sensing to improve contact-rich manipulation in unstructured environments.

Contribution

We introduce MagicGripper, a novel, integrated multimodal sensor system for robotic grippers, featuring mini-MagicTac for enhanced tactile and visual perception in a compact form.

Findings

High-resolution tactile feedback demonstrated

Effective contact localization and force regression achieved

Reliable performance across real-world conditions

Abstract

Contact-rich manipulation in unstructured environments demands precise, multimodal perception to enable robust and adaptive control. Vision-based tactile sensors (VBTSs) have emerged as an effective solution; however, conventional VBTSs often face challenges in achieving compact, multi-modal functionality due to hardware constraints and algorithmic complexity. In this work, we present MagicGripper, a multimodal sensor-integrated gripper designed for contact-rich robotic manipulation. Building on our prior design, MagicTac, we develop a compact variant, mini-MagicTac, which features a three-dimensional, multi-layered grid embedded in a soft elastomer. MagicGripper integrates mini-MagicTac, enabling high-resolution tactile feedback alongside proximity and visual sensing within a compact, gripper-compatible form factor. We conduct a thorough evaluation of mini-MagicTac's performance, demonstrating its capabilities in spatial resolution, contact localization, and force regression. We also assess its robustness across manufacturing variability, mechanical deformation, and sensing performance under real-world conditions. Furthermore, we validate the effectiveness of MagicGripper through three representative robotic tasks: a teleoperated assembly task, a contact-based alignment task, and an autonomous robotic grasping task. Across these experiments, MagicGripper exhibits reliable multimodal perception, accurate force estimation, and high adaptability to challenging manipulation scenarios. Our results highlight the potential of MagicGripper as a practical and versatile tool for embodied intelligence in complex, contact-rich environments.