OV-DEIM: Real-time DETR-Style Open-Vocabulary Object Detection with GridSynthetic Augmentation

TL;DR

OV-DEIM is a novel DETR-style open-vocabulary object detector that combines architectural innovations and a new data augmentation strategy to achieve state-of-the-art performance in real-time scenarios, especially for rare categories.

Contribution

The paper introduces OV-DEIM, a DETR-based open-vocabulary detector with a query supplement strategy and GridSynthetic augmentation, enhancing efficiency and accuracy over existing methods.

Findings

Achieves state-of-the-art open-vocabulary detection performance.

Improves detection of rare categories significantly.

Maintains real-time inference speed with architectural enhancements.

Abstract

Real-time open-vocabulary object detection (OVOD) is essential for practical deployment in dynamic environments, where models must recognize a large and evolving set of categories under strict latency constraints. Current real-time OVOD methods are predominantly built upon YOLO-style models. In contrast, real-time DETR-based methods still lag behind in terms of inference latency, model lightweightness, and overall performance. In this work, we present OV-DEIM, an end-to-end DETR-style open-vocabulary detector built upon the recent DEIMv2 framework with integrated vision-language modeling for efficient open-vocabulary inference. We further introduce a simple query supplement strategy that improves Fixed AP without compromising inference speed. Beyond architectural improvements, we introduce GridSynthetic, a simple yet effective data augmentation strategy that composes multiple training samples into structured image grids. By exposing the model to richer object co-occurrence patterns and spatial layouts within a single forward pass, GridSynthetic mitigates the negative impact of noisy localization signals on the classification loss and improves semantic discrimination, particularly for rare categories. Extensive experiments demonstrate that OV-DEIM achieves state-of-the-art performance on open-vocabulary detection benchmarks, delivering superior efficiency and notable improvements on challenging rare categories. Code and pretrained models are available at https://github.com/wleilei/OV-DEIM.