GeoLanG: Geometry-Aware Language-Guided Grasping with Unified RGB-D Multimodal Learning

TL;DR

GeoLanG is an end-to-end framework that unifies visual and linguistic inputs using CLIP, enhanced by depth-guided geometric priors and adaptive feature integration, to improve language-guided robotic grasping in cluttered and occluded scenes.

Contribution

The paper introduces GeoLanG, a novel geometry-aware, multimodal learning framework that effectively combines RGB-D data and language instructions for robust robotic grasping.

Findings

Achieves precise grasping in cluttered environments

Demonstrates robustness in occluded and low-texture scenes

Outperforms existing methods on OCID-VLG dataset

Abstract

Language-guided grasping has emerged as a promising paradigm for enabling robots to identify and manipulate target objects through natural language instructions, yet it remains highly challenging in cluttered or occluded scenes. Existing methods often rely on multi-stage pipelines that separate object perception and grasping, which leads to limited cross-modal fusion, redundant computation, and poor generalization in cluttered, occluded, or low-texture scenes. To address these limitations, we propose GeoLanG, an end-to-end multi-task framework built upon the CLIP architecture that unifies visual and linguistic inputs into a shared representation space for robust semantic alignment and improved generalization. To enhance target discrimination under occlusion and low-texture conditions, we explore a more effective use of depth information through the Depth-guided Geometric Module (DGGM), which converts depth into explicit geometric priors and injects them into the attention mechanism without additional computational overhead. In addition, we propose Adaptive Dense Channel Integration, which adaptively balances the contributions of multi-layer features to produce more discriminative and generalizable visual representations. Extensive experiments on the OCID-VLG dataset, as well as in both simulation and real-world hardware, demonstrate that GeoLanG enables precise and robust language-guided grasping in complex, cluttered environments, paving the way toward more reliable multimodal robotic manipulation in real-world human-centric settings.