SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection

TL;DR

SHiNe introduces a hierarchy-aware classifier for open-vocabulary object detection that leverages class hierarchies to improve robustness and accuracy across diverse vocabularies without additional inference costs.

Contribution

It proposes a novel, training-free hierarchy-based classifier that enhances open-vocabulary detection robustness by integrating semantic hierarchy knowledge into existing detectors.

Findings

Up to +31.9% mAP50 improvement with ground truth hierarchies.

Improves CLIP zero-shot accuracy by +2.8% on ImageNet-1k.

Seamlessly integrates with existing detectors without extra inference overhead.

Abstract

Open-vocabulary object detection (OvOD) has transformed detection into a language-guided task, empowering users to freely define their class vocabularies of interest during inference. However, our initial investigation indicates that existing OvOD detectors exhibit significant variability when dealing with vocabularies across various semantic granularities, posing a concern for real-world deployment. To this end, we introduce Semantic Hierarchy Nexus (SHiNe), a novel classifier that uses semantic knowledge from class hierarchies. It runs offline in three steps: i) it retrieves relevant super-/sub-categories from a hierarchy for each target class; ii) it integrates these categories into hierarchy-aware sentences; iii) it fuses these sentence embeddings to generate the nexus classifier vector. Our evaluation on various detection benchmarks demonstrates that SHiNe enhances robustness across diverse vocabulary granularities, achieving up to +31.9% mAP50 with ground truth hierarchies, while retaining improvements using hierarchies generated by large language models. Moreover, when applied to open-vocabulary classification on ImageNet-1k, SHiNe improves the CLIP zero-shot baseline by +2.8% accuracy. SHiNe is training-free and can be seamlessly integrated with any off-the-shelf OvOD detector, without incurring additional computational overhead during inference. The code is open source.