UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learning

TL;DR

This paper introduces UAV-Flow, a real-world benchmark for language-guided fine-grained UAV control, enabling UAVs to imitate expert trajectories based on atomic language instructions, advancing human-drone interaction.

Contribution

It presents the first real-world dataset, task formulation, and control framework for language-conditioned UAV trajectory imitation, bridging the sim-to-real gap.

Findings

VLA models outperform VLN baselines in UAV control tasks.

Spatial grounding is crucial for fine-grained language-conditioned UAV control.

UAV-Flow enables direct deployment without sim-to-real transfer issues.

Abstract

Unmanned Aerial Vehicles (UAVs) are evolving into language-interactive platforms, enabling more intuitive forms of human-drone interaction. While prior works have primarily focused on high-level planning and long-horizon navigation, we shift attention to language-guided fine-grained trajectory control, where UAVs execute short-range, reactive flight behaviors in response to language instructions. We formalize this problem as the Flying-on-a-Word (Flow) task and introduce UAV imitation learning as an effective approach. In this framework, UAVs learn fine-grained control policies by mimicking expert pilot trajectories paired with atomic language instructions. To support this paradigm, we present UAV-Flow, the first real-world benchmark for language-conditioned, fine-grained UAV control. It includes a task formulation, a large-scale dataset collected in diverse environments, a deployable control framework, and a simulation suite for systematic evaluation. Our design enables UAVs to closely imitate the precise, expert-level flight trajectories of human pilots and supports direct deployment without sim-to-real gap. We conduct extensive experiments on UAV-Flow, benchmarking VLN and VLA paradigms. Results show that VLA models are superior to VLN baselines and highlight the critical role of spatial grounding in the fine-grained Flow setting.