VK-Det: Visual Knowledge Guided Prototype Learning for Open-Vocabulary Aerial Object Detection

TL;DR

VK-Det is a novel framework for open-vocabulary aerial object detection that leverages visual knowledge and prototype-based pseudo-labeling to improve detection of unseen categories without extra supervision.

Contribution

It introduces a visual knowledge-guided approach with a prototype-aware pseudo-labeling strategy, enabling better open-vocabulary detection without additional supervision.

Findings

Achieves state-of-the-art mAP on DIOR and DOTA datasets.

Outperforms supervised methods in open-vocabulary detection.

Utilizes intrinsic visual features for fine-grained localization.

Abstract

To identify objects beyond predefined categories, open-vocabulary aerial object detection (OVAD) leverages the zero-shot capabilities of visual-language models (VLMs) to generalize from base to novel categories. Existing approaches typically utilize self-learning mechanisms with weak text supervision to generate region-level pseudo-labels to align detectors with VLMs semantic spaces. However, text dependence induces semantic bias, restricting open-vocabulary expansion to text-specified concepts. We propose $\textbf{VK-Det}$, a $\textbf{V}$isual $\textbf{K}$nowledge-guided open-vocabulary object $\textbf{Det}$ection framework $\textit{without}$ extra supervision. First, we discover and leverage vision encoder's inherent informative region perception to attain fine-grained localization and adaptive distillation. Second, we introduce a novel prototype-aware pseudo-labeling strategy. It models inter-class decision boundaries through feature clustering and maps detection regions to latent categories via prototype matching. This enhances attention to novel objects while compensating for missing supervision. Extensive experiments show state-of-the-art performance, achieving 30.1 $\mathrm{mAP}^{N}$ on DIOR and 23.3 $\mathrm{mAP}^{N}$ on DOTA, outperforming even extra supervised methods.