OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective

TL;DR

OccuFly introduces a novel camera-based aerial semantic scene completion benchmark, addressing the lack of LiDAR-free datasets and revealing current vision models' limitations in aerial 3D perception.

Contribution

The paper presents the first real-world, camera-based aerial SSC benchmark, OccuFly, with extensive data and benchmarks, and proposes a LiDAR-free data generation framework for aerial environments.

Findings

Current vision models show limitations in aerial SSC tasks.

OccuFly provides over 20,000 samples across diverse environments.

Benchmarking reveals fundamental challenges in aerial 3D scene understanding.

Abstract

Semantic Scene Completion (SSC) is essential for 3D perception in mobile robotics, as it enables holistic scene understanding by jointly estimating dense volumetric occupancy and per-voxel semantics. Although SSC has been widely studied in terrestrial domains such as autonomous driving, aerial settings like autonomous flying remain largely unexplored, thereby limiting progress on downstream applications. Furthermore, LiDAR sensors are the primary modality for SSC data generation, which poses challenges for most uncrewed aerial vehicles (UAVs) due to flight regulations, mass and energy constraints, and the sparsity of LiDAR point clouds from elevated viewpoints. To address these limitations, we propose a LiDAR-free, camera-based data generation framework. By leveraging classical 3D reconstruction, our framework automates semantic label transfer by lifting <10% of annotated images into the reconstructed point cloud, substantially minimizing manual 3D annotation effort. Based on this framework, we introduce OccuFly, the first real-world, camera-based aerial SSC benchmark, captured across multiple altitudes and all seasons. OccuFly provides over 20,000 samples of images, semantic voxel grids, and metric depth maps across 21 semantic classes in urban, industrial, and rural environments, and follows established data organization for seamless integration. We benchmark both SSC and metric monocular depth estimation on OccuFly, revealing fundamental limitations of current vision foundation models in aerial settings and establishing new challenges for robust 3D scene understanding in the aerial domain. Visit https://github.com/markus-42/occufly.