Efficient Decoder-free Object Detection with Transformers

TL;DR

This paper introduces a novel decoder-free transformer-based object detector that simplifies the detection process, achieves high efficiency, and outperforms existing models in accuracy and computational cost on the MS COCO benchmark.

Contribution

The paper proposes a decoder-free, encoder-only transformer architecture for object detection, reducing training time and computational cost while maintaining high accuracy.

Findings

Outperforms DETR by 2.5% AP with 28% less computation and over 10x fewer training epochs.

Achieves over 5.5% AP gain compared to RetinaNet while reducing 70% of computation.

Demonstrates high efficiency and accuracy on the MS COCO benchmark.

Abstract

Vision transformers (ViTs) are changing the landscape of object detection approaches. A natural usage of ViTs in detection is to replace the CNN-based backbone with a transformer-based backbone, which is straightforward and effective, with the price of bringing considerable computation burden for inference. More subtle usage is the DETR family, which eliminates the need for many hand-designed components in object detection but introduces a decoder demanding an extra-long time to converge. As a result, transformer-based object detection can not prevail in large-scale applications. To overcome these issues, we propose a novel decoder-free fully transformer-based (DFFT) object detector, achieving high efficiency in both training and inference stages, for the first time. We simplify objection detection into an encoder-only single-level anchor-based dense prediction problem by centering around two entry points: 1) Eliminate the training-inefficient decoder and leverage two strong encoders to preserve the accuracy of single-level feature map prediction; 2) Explore low-level semantic features for the detection task with limited computational resources. In particular, we design a novel lightweight detection-oriented transformer backbone that efficiently captures low-level features with rich semantics based on a well-conceived ablation study. Extensive experiments on the MS COCO benchmark demonstrate that DFFT_SMALL outperforms DETR by 2.5% AP with 28% computation cost reduction and more than $10$x fewer training epochs. Compared with the cutting-edge anchor-based detector RetinaNet, DFFT_SMALL obtains over 5.5% AP gain while cutting down 70% computation cost.