OS-W2S: An Automatic Labeling Engine for Language-Guided Open-Set Aerial Object Detection

TL;DR

This paper introduces OS-W2S, an automatic labeling engine for language-guided open-set aerial object detection, and constructs a large-scale dataset MI-OAD to improve fine-grained detection and grounding in aerial imagery.

Contribution

The paper presents a novel automatic annotation pipeline and a large-scale dataset for language-guided open-set aerial detection, enabling significant performance improvements and zero-shot transfer capabilities.

Findings

MI-OAD contains 163,023 images and 2 million captions, 40 times larger than comparable datasets.

Training on MI-OAD improves AP50 by +31.1 and Recall@10 by +34.7 for zero-shot detection.

Pre-training on MI-OAD achieves state-of-the-art results on multiple benchmarks.

Abstract

In recent years, language-guided open-set aerial object detection has gained significant attention due to its better alignment with real-world application needs. However, due to limited datasets, most existing language-guided methods primarily focus on vocabulary-level descriptions, which fail to meet the demands of fine-grained open-world detection. To address this limitation, we propose constructing a large-scale language-guided open-set aerial detection dataset, encompassing three levels of language guidance: from words to phrases, and ultimately to sentences. Centered around an open-source large vision-language model and integrating image-operation-based preprocessing with BERT-based postprocessing, we present the OS-W2S Label Engine, an automatic annotation pipeline capable of handling diverse scene annotations for aerial images. Using this label engine, we expand existing aerial detection datasets with rich textual annotations and construct a novel benchmark dataset, called MI-OAD, addressing the limitations of current remote sensing grounding data and enabling effective language-guided open-set aerial detection. Specifically, MI-OAD contains 163,023 images and 2 million image-caption pairs, approximately 40 times larger than comparable datasets. To demonstrate the effectiveness and quality of MI-OAD, we evaluate three representative tasks. On language-guided open-set aerial detection, training on MI-OAD lifts Grounding DINO by +31.1 AP$_{50}$ and +34.7 Recall@10 with sentence-level inputs under zero-shot transfer. Moreover, using MI-OAD for pre-training yields state-of-the-art performance on multiple existing open-vocabulary aerial detection and remote sensing visual grounding benchmarks, validating both the effectiveness of the dataset and the high quality of its OS-W2S annotations. More details are available at https://github.com/GT-Wei/MI-OAD.