Grasp What You Want: Embodied Dexterous Grasping System Driven by Your Voice

TL;DR

This paper presents EDGS, a novel voice-driven robotic grasping system that integrates vision-language models and human-inspired grasping strategies to improve object interaction in cluttered environments.

Contribution

The paper introduces a new embodied grasping system that combines semantic-object alignment with advanced manipulation techniques driven by voice commands.

Findings

High success rate in complex grasping tasks

Effective integration of voice commands with visual data

Robust and precise grasping strategy demonstrated

Abstract

In recent years, as robotics has advanced, human-robot collaboration has gained increasing importance. However, current robots struggle to fully and accurately interpret human intentions from voice commands alone. Traditional gripper and suction systems often fail to interact naturally with humans, lack advanced manipulation capabilities, and are not adaptable to diverse tasks, especially in unstructured environments. This paper introduces the Embodied Dexterous Grasping System (EDGS), designed to tackle object grasping in cluttered environments for human-robot interaction. We propose a novel approach to semantic-object alignment using a Vision-Language Model (VLM) that fuses voice commands and visual information, significantly enhancing the alignment of multi-dimensional attributes of target objects in complex scenarios. Inspired by human hand-object interactions, we develop a robust, precise, and efficient grasping strategy, incorporating principles like the thumb-object axis, multi-finger wrapping, and fingertip interaction with an object's contact mechanics. We also design experiments to assess Referring Expression Representation Enrichment (RERE) in referring expression segmentation, demonstrating that our system accurately detects and matches referring expressions. Extensive experiments confirm that EDGS can effectively handle complex grasping tasks, achieving stability and high success rates, highlighting its potential for further development in the field of Embodied AI.