Revisiting DETR Pre-training for Object Detection

TL;DR

This paper critically evaluates self-supervised pre-training methods for DETR-based object detection, introduces an improved self-training approach, and demonstrates significant performance gains on COCO and PASCAL VOC benchmarks.

Contribution

It identifies limitations of existing pre-training methods like DETReg, proposes Simple Self-training, and shows how synthetic datasets further boost detection accuracy.

Findings

Simple Self-training improves AP scores on COCO.

Synthetic datasets from image-to-text and text-to-image models enhance detection.

Achieved 59.3% AP on COCO val set, surpassing previous methods.

Abstract

Motivated by the remarkable achievements of DETR-based approaches on COCO object detection and segmentation benchmarks, recent endeavors have been directed towards elevating their performance through self-supervised pre-training of Transformers while preserving a frozen backbone. Noteworthy advancements in accuracy have been documented in certain studies. Our investigation delved deeply into a representative approach, DETReg, and its performance assessment in the context of emerging models like $\mathcal{H}$-Deformable-DETR. Regrettably, DETReg proves inadequate in enhancing the performance of robust DETR-based models under full data conditions. To dissect the underlying causes, we conduct extensive experiments on COCO and PASCAL VOC probing elements such as the selection of pre-training datasets and strategies for pre-training target generation. By contrast, we employ an optimized approach named Simple Self-training which leads to marked enhancements through the combination of an improved box predictor and the Objects$365$ benchmark. The culmination of these endeavors results in a remarkable AP score of $59.3\%$ on the COCO val set, outperforming $\mathcal{H}$-Deformable-DETR + Swin-L without pre-training by $1.4\%$. Moreover, a series of synthetic pre-training datasets, generated by merging contemporary image-to-text(LLaVA) and text-to-image (SDXL) models, significantly amplifies object detection capabilities.