UAVScenes: A Multi-Modal Dataset for UAVs

TL;DR

UAVScenes is a comprehensive multi-modal UAV dataset with frame-wise semantic annotations for images and LiDAR, enabling advanced perception tasks like segmentation, localization, and view synthesis.

Contribution

The paper introduces UAVScenes, a large-scale, multi-modal UAV dataset with detailed annotations for diverse high-level perception tasks, filling a critical gap in existing UAV datasets.

Findings

Provides annotations for both images and LiDAR point clouds

Enables benchmarking of multiple perception tasks

Supports high-level scene understanding in UAV applications

Abstract

Multi-modal perception is essential for unmanned aerial vehicle (UAV) operations, as it enables a comprehensive understanding of the UAVs' surrounding environment. However, most existing multi-modal UAV datasets are primarily biased toward localization and 3D reconstruction tasks, or only support map-level semantic segmentation due to the lack of frame-wise annotations for both camera images and LiDAR point clouds. This limitation prevents them from being used for high-level scene understanding tasks. To address this gap and advance multi-modal UAV perception, we introduce UAVScenes, a large-scale dataset designed to benchmark various tasks across both 2D and 3D modalities. Our benchmark dataset is built upon the well-calibrated multi-modal UAV dataset MARS-LVIG, originally developed only for simultaneous localization and mapping (SLAM). We enhance this dataset by providing manually labeled semantic annotations for both frame-wise images and LiDAR point clouds, along with accurate 6-degree-of-freedom (6-DoF) poses. These additions enable a wide range of UAV perception tasks, including segmentation, depth estimation, 6-DoF localization, place recognition, and novel view synthesis (NVS). Our dataset is available at https://github.com/sijieaaa/UAVScenes