Towards Robust Cross-Dataset Object Detection Generalization under Domain Specificity

TL;DR

This paper investigates how object detectors perform across different datasets, revealing that transferability is highly dependent on setting similarity, with domain shift being the main challenge in cross-dataset detection.

Contribution

It provides a detailed analysis of cross-dataset object detection under setting specificity, highlighting the impact of domain shift and proposing evaluation protocols to better understand generalization.

Findings

Transfer within same setting type is stable.

Transfer across different setting types drops significantly.

Domain shift dominates in the hardest transfer regimes.

Abstract

Object detectors often perform well in-distribution, yet degrade sharply on a different benchmark. We study cross-dataset object detection (CD-OD) through a lens of setting specificity. We group benchmarks into setting-agnostic datasets with diverse everyday scenes and setting-specific datasets tied to a narrow environment, and evaluate a standard detector family across all train--test pairs. This reveals a clear structure in CD-OD: transfer within the same setting type is relatively stable, while transfer across setting types drops substantially and is often asymmetric. The most severe breakdowns occur when transferring from specific sources to agnostic targets, and persist after open-label alignment, indicating that domain shift dominates in the hardest regimes. To disentangle domain shift from label mismatch, we compare closed-label transfer with an open-label protocol that maps predicted classes to the nearest target label using CLIP similarity. Open-label evaluation yields consistent but bounded gains, and many corrected cases correspond to semantic near-misses supported by the image evidence. Overall, we provide a principled characterization of CD-OD under setting specificity and practical guidance for evaluating detectors under distribution shift. Code will be released at \href{[https://github.com/Ritabrata04/cdod-icpr.git}{https://github.com/Ritabrata04/cdod-icpr}.