QwenGrasp: A Usage of Large Vision-Language Model for Target-Oriented Grasping

TL;DR

QwenGrasp leverages a large vision-language model combined with a neural network to enable robots to understand natural language instructions and perform precise target-oriented 6-DoF grasping in unstructured scenes, improving safety and accuracy.

Contribution

This paper introduces QwenGrasp, a novel model integrating vision-language understanding with grasping neural networks for flexible, instruction-based robotic grasping.

Findings

QwenGrasp accurately grasps target objects based on vague or descriptive instructions.

The model can suspend tasks and provide feedback when instructions are infeasible or irrelevant.

QwenGrasp demonstrates superior comprehension of human intentions in complex scenes.

Abstract

Target-oriented grasping in unstructured scenes with language control is essential for intelligent robot arm grasping. The ability for the robot arm to understand the human language and execute corresponding grasping actions is a pivotal challenge. In this paper, we propose a combination model called QwenGrasp which combines a large vision-language model with a 6-DoF grasp neural network. QwenGrasp is able to conduct a 6-DoF grasping task on the target object with textual language instruction. We design a complete experiment with six-dimension instructions to test the QwenGrasp when facing with different cases. The results show that QwenGrasp has a superior ability to comprehend the human intention. Even in the face of vague instructions with descriptive words or instructions with direction information, the target object can be grasped accurately. When QwenGrasp accepts the instruction which is not feasible or not relevant to the grasping task, our approach has the ability to suspend the task execution and provide a proper feedback to humans, improving the safety. In conclusion, with the great power of large vision-language model, QwenGrasp can be applied in the open language environment to conduct the target-oriented grasping task with freely input instructions.