Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Head

TL;DR

This paper introduces OmDet-Turbo, a real-time transformer-based open-vocabulary object detection model with an efficient fusion head, achieving high speed and competitive accuracy on benchmark datasets.

Contribution

The paper presents OmDet-Turbo, a novel real-time open-vocabulary detection model with an innovative fusion head that significantly improves inference speed and maintains high detection performance.

Findings

OmDet-Turbo achieves 100.2 FPS with TensorRT and language cache.

It performs nearly on par with state-of-the-art models in zero-shot detection.

Sets new benchmarks on ODinW and OVDEval datasets.

Abstract

End-to-end transformer-based detectors (DETRs) have shown exceptional performance in both closed-set and open-vocabulary object detection (OVD) tasks through the integration of language modalities. However, their demanding computational requirements have hindered their practical application in real-time object detection (OD) scenarios. In this paper, we scrutinize the limitations of two leading models in the OVDEval benchmark, OmDet and Grounding-DINO, and introduce OmDet-Turbo. This novel transformer-based real-time OVD model features an innovative Efficient Fusion Head (EFH) module designed to alleviate the bottlenecks observed in OmDet and Grounding-DINO. Notably, OmDet-Turbo-Base achieves a 100.2 frames per second (FPS) with TensorRT and language cache techniques applied. Notably, in zero-shot scenarios on COCO and LVIS datasets, OmDet-Turbo achieves performance levels nearly on par with current state-of-the-art supervised models. Furthermore, it establishes new state-of-the-art benchmarks on ODinW and OVDEval, boasting an AP of 30.1 and an NMS-AP of 26.86, respectively. The practicality of OmDet-Turbo in industrial applications is underscored by its exceptional performance on benchmark datasets and superior inference speed, positioning it as a compelling choice for real-time object detection tasks. Code: \url{https://github.com/om-ai-lab/OmDet}