DetRefiner: Model-Agnostic Detection Refinement with Feature Fusion Transformer

TL;DR

DetRefiner is a plug-and-play framework that fuses global and local features via a lightweight Transformer to improve open-vocabulary object detection, enhancing performance without retraining base models.

Contribution

It introduces a model-agnostic, training-free method to refine detection confidence by integrating global and local contextual cues using feature fusion transformers.

Findings

Achieves up to +10.1 AP improvement on novel categories.

Enhances multiple OVOD models across datasets like COCO and LVIS.

Operates solely on base detector predictions without retraining.

Abstract

Open-vocabulary object detection (OVOD) aims to detect both seen and unseen categories, yet existing methods often struggle to generalize to novel objects due to limited integration of global and local contextual cues. We propose DetRefiner, a simple yet effective plug-and-play framework that learns to fuse global and local features to refine open-vocabulary detection. DetRefiner processes global image features and patch-level image features from foundational models (e.g., DINOv3) through a lightweight Transformer encoder. The encoder produces a class vector capturing image-level attributes and patch vectors representing local region attributes, from which attribute reliability is inferred to recalibrate the base model's confidence. Notably, DetRefiner is trained independently of the base OVOD model, requiring neither access to its internal features nor retraining. At inference, it operates solely on the base detector's predictions, producing auxiliary calibration scores that are merged with the base detector's scores to yield the final refined confidence. Despite this simplicity, DetRefiner consistently enhances multiple OVOD models across COCO, LVIS, ODinW13, and Pascal VOC, achieving gains of up to +10.1 AP on novel categories. These results highlight that learning to fuse global and local representations offers a powerful and general mechanism for advancing open-world object detection. Our codes and models are available at https://github.com/hitachi-rd-cv/detrefiner.