DETR Doesn't Need Multi-Scale or Locality Design

TL;DR

This paper introduces a simplified DETR object detector that forgoes multi-scale features and locality biases, yet achieves competitive accuracy through innovative position bias and backbone pre-training techniques.

Contribution

It demonstrates that a plain single-scale DETR can match state-of-the-art performance using novel position bias and pre-training strategies.

Findings

Achieved 63.9 mAP on Object365 with Swin-L backbone.

Simple techniques effectively compensate for lack of multi-scale features.

Plain DETR rivals complex multi-scale detectors in accuracy.

Abstract

This paper presents an improved DETR detector that maintains a "plain" nature: using a single-scale feature map and global cross-attention calculations without specific locality constraints, in contrast to previous leading DETR-based detectors that reintroduce architectural inductive biases of multi-scale and locality into the decoder. We show that two simple technologies are surprisingly effective within a plain design to compensate for the lack of multi-scale feature maps and locality constraints. The first is a box-to-pixel relative position bias (BoxRPB) term added to the cross-attention formulation, which well guides each query to attend to the corresponding object region while also providing encoding flexibility. The second is masked image modeling (MIM)-based backbone pre-training which helps learn representation with fine-grained localization ability and proves crucial for remedying dependencies on the multi-scale feature maps. By incorporating these technologies and recent advancements in training and problem formation, the improved "plain" DETR showed exceptional improvements over the original DETR detector. By leveraging the Object365 dataset for pre-training, it achieved 63.9 mAP accuracy using a Swin-L backbone, which is highly competitive with state-of-the-art detectors which all heavily rely on multi-scale feature maps and region-based feature extraction. Code is available at https://github.com/impiga/Plain-DETR .