Hierarchical LLMs In-the-Loop Optimization for Real-Time Multi-Robot Target Tracking under Unknown Hazards

TL;DR

This paper introduces a hierarchical optimization framework that integrates Large Language Models into multi-robot coordination, enabling real-time adaptation to hazards and improving safety in dynamic environments.

Contribution

The paper presents a novel bi-level optimization approach that leverages LLMs for task configuration and hazard reasoning in multi-robot target tracking.

Findings

Effective real-time hazard adaptation demonstrated in simulations.

Successful validation in real-world multi-robot experiments.

Enhanced safety and flexibility through hierarchical LLM integration.

Abstract

Real-time multi-robot coordination in hazardous and adversarial environments requires fast, reliable adaptation to dynamic threats. While Large Language Models (LLMs) offer strong high-level reasoning capabilities, the lack of safety guarantees limits their direct use in critical decision-making. In this paper, we propose a hierarchical optimization framework that integrates LLMs into the decision loop for multi-robot target tracking in dynamic and hazardous environments. Rather than generating control actions directly, LLMs are used to generate task configuration and adjust parameters in a bi-level task allocation and planning problem. We formulate multi-robot coordination for tracking tasks as a bi-level optimization problem, with LLMs to reason about potential hazards in the environment and the status of the robot team and modify both the inner and outer levels of the optimization. This hierarchical approach enables real-time adjustments to the robots' behavior. Additionally, a human supervisor can offer broad guidance and assessments to address unexpected dangers, model mismatches, and performance issues arising from local minima. We validate our proposed framework in both simulation and real-world experiments with comprehensive evaluations, demonstrating its effectiveness and showcasing its capability for safe LLM integration for multi-robot systems.