A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping

TL;DR

This paper introduces a parameter-efficient CLIP-based framework for language-guided object grounding and robot grasping, improving performance while reducing computational demands for real-world robotic applications.

Contribution

It proposes a novel bi-directional vision-language adapter and depth fusion branch, enabling effective multimodal understanding with fewer parameters compared to full-model tuning.

Findings

Outperforms existing CLIP-based methods in object grounding accuracy

Successfully interprets object attributes from simple language descriptions

Demonstrates strong spatial reasoning in complex scenarios

Abstract

The language-guided robot grasping task requires a robot agent to integrate multimodal information from both visual and linguistic inputs to predict actions for target-driven grasping. While recent approaches utilizing Multimodal Large Language Models (MLLMs) have shown promising results, their extensive computation and data demands limit the feasibility of local deployment and customization. To address this, we propose a novel CLIP-based multimodal parameter-efficient tuning (PET) framework designed for three language-guided object grounding and grasping tasks: (1) Referring Expression Segmentation (RES), (2) Referring Grasp Synthesis (RGS), and (3) Referring Grasp Affordance (RGA). Our approach introduces two key innovations: a bi-directional vision-language adapter that aligns multimodal inputs for pixel-level language understanding and a depth fusion branch that incorporates geometric cues to facilitate robot grasping predictions. Experiment results demonstrate superior performance in the RES object grounding task compared with existing CLIP-based full-model tuning or PET approaches. In the RGS and RGA tasks, our model not only effectively interprets object attributes based on simple language descriptions but also shows strong potential for comprehending complex spatial reasoning scenarios, such as multiple identical objects present in the workspace. Project page: https://z.umn.edu/etog-etrg