MM-UAVBench: How Well Do Multimodal Large Language Models See, Think, and Plan in Low-Altitude UAV Scenarios?

TL;DR

This paper introduces MM-UAVBench, a comprehensive benchmark for evaluating multimodal large language models in low-altitude UAV scenarios, highlighting current models' limitations and guiding future research.

Contribution

The paper presents MM-UAVBench, the first benchmark systematically assessing MLLMs' perception, cognition, and planning in UAV contexts with real-world data.

Findings

Current MLLMs struggle with complex UAV visual and cognitive tasks.

Identified bottlenecks include spatial bias and multi-view understanding.

Benchmark facilitates targeted improvements in UAV-related MLLMs.

Abstract

While Multimodal Large Language Models (MLLMs) have exhibited remarkable general intelligence across diverse domains, their potential in low-altitude applications dominated by Unmanned Aerial Vehicles (UAVs) remains largely underexplored. Existing MLLM benchmarks rarely cover the unique challenges of low-altitude scenarios, while UAV-related evaluations mainly focus on specific tasks such as localization or navigation, without a unified evaluation of MLLMs'general intelligence. To bridge this gap, we present MM-UAVBench, a comprehensive benchmark that systematically evaluates MLLMs across three core capability dimensions-perception, cognition, and planning-in low-altitude UAV scenarios. MM-UAVBench comprises 19 sub-tasks with over 5.7K manually annotated questions, all derived from real-world UAV data collected from public datasets. Extensive experiments on 16 open-source and proprietary MLLMs reveal that current models struggle to adapt to the complex visual and cognitive demands of low-altitude scenarios. Our analyses further uncover critical bottlenecks such as spatial bias and multi-view understanding that hinder the effective deployment of MLLMs in UAV scenarios. We hope MM-UAVBench will foster future research on robust and reliable MLLMs for real-world UAV intelligence.