Large Language Model-Driven Closed-Loop UAV Operation with Semantic Observations

TL;DR

This paper introduces a closed-loop framework utilizing large language models for UAV control, transforming sensor data into semantic descriptions to improve reliability and safety through iterative feedback and simulation-based refinement.

Contribution

The paper presents a novel LLM-driven UAV operation framework with semantic observations and a feedback loop, enhancing reliability and safety in complex IoT environments.

Findings

Significantly higher success rates compared to baseline methods.

Effective transformation of numerical data into semantic descriptions.

Robust performance across various UAV control task complexities.

Abstract

Recent advances in large Language Models (LLMs) have revolutionized mobile robots, including unmanned aerial vehicles (UAVs), enabling their intelligent operation within Internet of Things (IoT) ecosystems. However, LLMs still face challenges from logical reasoning and complex decision-making, leading to concerns about the reliability of LLM-driven UAV operations in IoT applications. In this paper, we propose a closed-loop LLM-driven UAV operation code generation framework that enables reliable UAV operations powered by effective feedback and refinement using two LLM modules, i.e., a Code Generator and an Evaluator. Our framework transforms numerical state observations from UAV operations into semantic trajectory descriptions to enhance the evaluator LLM's understanding of UAV dynamics for precise feedback generation. Our framework also enables a simulation-based refinement process, and hence eliminates the risks to physical UAVs caused by incorrect code execution during the refinement. Extensive experiments on UAV control tasks with different complexities are conducted. The experimental results show that our framework can achieve reliable UAV operations using LLMs, which significantly outperforms baseline methods in terms of success rate and completeness with the increase of task complexity.