Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigation

TL;DR

This paper introduces SpatialSky-Bench, a comprehensive benchmark for evaluating the spatial intelligence of Vision-Language Models in UAV navigation, revealing current limitations and proposing a new specialized VLM, Sky-VLM, that achieves state-of-the-art results.

Contribution

The paper presents a new benchmark, SpatialSky-Bench, and a large dataset, SpatialSky-Dataset, along with a specialized VLM, Sky-VLM, tailored for UAV spatial reasoning tasks.

Findings

Mainstream VLMs perform poorly in complex UAV scenarios.

Sky-VLM outperforms existing models across all benchmark tasks.

The benchmark reveals significant gaps in current VLM spatial capabilities.

Abstract

Vision-Language Models (VLMs), leveraging their powerful visual perception and reasoning capabilities, have been widely applied in Unmanned Aerial Vehicle (UAV) tasks. However, the spatial intelligence capabilities of existing VLMs in UAV scenarios remain largely unexplored, raising concerns about their effectiveness in navigating and interpreting dynamic environments. To bridge this gap, we introduce SpatialSky-Bench, a comprehensive benchmark specifically designed to evaluate the spatial intelligence capabilities of VLMs in UAV navigation. Our benchmark comprises two categories-Environmental Perception and Scene Understanding-divided into 13 subcategories, including bounding boxes, color, distance, height, and landing safety analysis, among others. Extensive evaluations of various mainstream open-source and closed-source VLMs reveal unsatisfactory performance in complex UAV navigation scenarios, highlighting significant gaps in their spatial capabilities. To address this challenge, we developed the SpatialSky-Dataset, a comprehensive dataset containing 1M samples with diverse annotations across various scenarios. Leveraging this dataset, we introduce Sky-VLM, a specialized VLM designed for UAV spatial reasoning across multiple granularities and contexts. Extensive experimental results demonstrate that Sky-VLM achieves state-of-the-art performance across all benchmark tasks, paving the way for the development of VLMs suitable for UAV scenarios. The source code is available at https://github.com/linglingxiansen/SpatialSKy.