Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System

TL;DR

This paper introduces a hierarchical multimodal framework combining large language models and vision-language models to enable robust semantic navigation and manipulation in heterogeneous aerial-ground robotic systems, demonstrating adaptability and coordination in dynamic environments.

Contribution

It presents the first integrated aerial-ground robotic system using VLM and LLM for high-level reasoning, perception, and control, enhancing generalizability and task performance.

Findings

Successful real-world validation on long-horizon object tasks.

Zero-shot adaptability to new scenarios.

Enhanced spatial accuracy with GridMask for manipulation.

Abstract

Heterogeneous multirobot systems show great potential in complex tasks requiring coordinated hybrid cooperation. However, existing methods that rely on static or task-specific models often lack generalizability across diverse tasks and dynamic environments. This highlights the need for generalizable intelligence that can bridge high-level reasoning with low-level execution across heterogeneous agents. To address this, we propose a hierarchical multimodal framework that integrates a prompted large language model (LLM) with a fine-tuned vision-language model (VLM). At the system level, the LLM performs hierarchical task decomposition and constructs a global semantic map, while the VLM provides semantic perception and object localization, where the proposed GridMask significantly enhances the VLM's spatial accuracy for reliable fine-grained manipulation. The aerial robot leverages this global map to generate semantic paths and guide the ground robot's local navigation and manipulation, ensuring robust coordination even in target-absent or ambiguous scenarios. We validate the framework through extensive simulation and real-world experiments on long-horizon object arrangement tasks, demonstrating zero-shot adaptability, robust semantic navigation, and reliable manipulation in dynamic environments. To the best of our knowledge, this work presents the first heterogeneous aerial-ground robotic system that integrates VLM-based perception with LLM-driven reasoning for global high-level task planning and execution.