Large Language Models to Enhance Multi-task Drone Operations in Simulated Environments

TL;DR

This paper presents a novel approach using fine-tuned large language models to enable natural language control of drones in simulated environments, improving accessibility and operational efficiency.

Contribution

It introduces a method integrating CodeT5 with AirSim to translate natural language commands into executable drone code, enhancing multi-task drone operations in simulation.

Findings

Superior task execution efficiency in simulations

Enhanced command understanding capabilities

Flexible environment construction in AirSim

Abstract

Benefiting from the rapid advancements in large language models (LLMs), human-drone interaction has reached unprecedented opportunities. In this paper, we propose a method that integrates a fine-tuned CodeT5 model with the Unreal Engine-based AirSim drone simulator to efficiently execute multi-task operations using natural language commands. This approach enables users to interact with simulated drones through prompts or command descriptions, allowing them to easily access and control the drone's status, significantly lowering the operational threshold. In the AirSim simulator, we can flexibly construct visually realistic dynamic environments to simulate drone applications in complex scenarios. By combining a large dataset of (natural language, program code) command-execution pairs generated by ChatGPT with developer-written drone code as training data, we fine-tune the CodeT5 to achieve automated translation from natural language to executable code for drone tasks. Experimental results demonstrate that the proposed method exhibits superior task execution efficiency and command understanding capabilities in simulated environments. In the future, we plan to extend the model functionality in a modular manner, enhancing its adaptability to complex scenarios and driving the application of drone technologies in real-world environments.