RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision

TL;DR

RT-DETRv3 introduces hierarchical dense positive supervision and novel training strategies to enhance real-time transformer-based object detection, significantly improving accuracy while maintaining efficiency.

Contribution

The paper proposes a hierarchical dense supervision method and a new learning strategy for RT-DETR, boosting training quality and detection performance in real-time object detection.

Findings

RT-DETRv3 outperforms existing real-time detectors on COCO.

RT-DETRv3-R18 achieves 48.1% AP, surpassing previous models.

RT-DETRv3-R101 reaches 54.6% AP, outperforming YOLOv10-X.

Abstract

RT-DETR is the first real-time end-to-end transformer-based object detector. Its efficiency comes from the framework design and the Hungarian matching. However, compared to dense supervision detectors like the YOLO series, the Hungarian matching provides much sparser supervision, leading to insufficient model training and difficult to achieve optimal results. To address these issues, we proposed a hierarchical dense positive supervision method based on RT-DETR, named RT-DETRv3. Firstly, we introduce a CNN-based auxiliary branch that provides dense supervision that collaborates with the original decoder to enhance the encoder feature representation. Secondly, to address insufficient decoder training, we propose a novel learning strategy involving self-attention perturbation. This strategy diversifies label assignment for positive samples across multiple query groups, thereby enriching positive supervisions. Additionally, we introduce a shared-weight decoder branch for dense positive supervision to ensure more high-quality queries matching each ground truth. Notably, all aforementioned modules are training-only. We conduct extensive experiments to demonstrate the effectiveness of our approach on COCO val2017. RT-DETRv3 significantly outperforms existing real-time detectors, including the RT-DETR series and the YOLO series. For example, RT-DETRv3-R18 achieves 48.1% AP (+1.6%/+1.4%) compared to RT-DETR-R18/RT-DETRv2-R18, while maintaining the same latency. Furthermore, RT-DETRv3-R101 can attain an impressive 54.6% AP outperforming YOLOv10-X. The code will be released at https://github.com/clxia12/RT-DETRv3.