HUGE-Bench: A Benchmark for High-Level UAV Vision-Language-Action Tasks

TL;DR

HUGE-Bench is a comprehensive high-level UAV vision-language-action benchmark with real-world scenes, designed to evaluate safety, process fidelity, and semantic understanding in complex UAV tasks.

Contribution

It introduces a new benchmark with high-level tasks, real-world digital twin scenes, and collision-aware evaluation metrics for UAV vision-language navigation.

Findings

Current models show significant gaps in semantic understanding.

Models struggle with safe and process-oriented UAV navigation.

HUGE-Bench provides a diagnostic platform for high-level UAV autonomy.

Abstract

Existing UAV vision-language navigation (VLN) benchmarks have enabled language-guided flight, but they largely focus on long, step-wise route descriptions with goal-centric evaluation, making them less diagnostic for real operations where brief, high-level commands must be grounded into safe multi-stage behaviors. We present HUGE-Bench, a benchmark for High-Level UAV Vision-Language-Action (HL-VLA) tasks that tests whether an agent can interpret concise language and execute complex, process-oriented trajectories with safety awareness. HUGE-Bench comprises 4 real-world digital twin scenes, 8 high-level tasks, and 2.56M meters of trajectories, and is built on an aligned 3D Gaussian Splatting (3DGS)-Mesh representation that combines photorealistic rendering with collision-capable geometry for scalable generation and collision-aware evaluation. We introduce process-oriented and collision-aware metrics to assess process fidelity, terminal accuracy, and safety. Experiments on representative state-of-the-art VLA models reveal significant gaps in high-level semantic completion and safe execution, highlighting HUGE-Bench as a diagnostic testbed for high-level UAV autonomy.