Recurrent Glimpse-based Decoder for Detection with Transformer

TL;DR

This paper introduces REGO, a recurrent glimpse-based decoder that improves DETR object detection by focusing attention on foreground objects through multi-stage processing, significantly reducing training epochs needed for high performance.

Contribution

The paper proposes a novel recurrent glimpse-based decoder (REGO) that enhances DETR detection accuracy and training efficiency by iterative focus on regions of interest.

Findings

REGO achieves 44.8 AP on MSCOCO with only 36 epochs.

REGO boosts DETR performance by up to 7% at 50 epochs.

REGO can be integrated into existing DETR variants without disrupting end-to-end training.

Abstract

Although detection with Transformer (DETR) is increasingly popular, its global attention modeling requires an extremely long training period to optimize and achieve promising detection performance. Alternative to existing studies that mainly develop advanced feature or embedding designs to tackle the training issue, we point out that the Region-of-Interest (RoI) based detection refinement can easily help mitigate the difficulty of training for DETR methods. Based on this, we introduce a novel REcurrent Glimpse-based decOder (REGO) in this paper. In particular, the REGO employs a multi-stage recurrent processing structure to help the attention of DETR gradually focus on foreground objects more accurately. In each processing stage, visual features are extracted as glimpse features from RoIs with enlarged bounding box areas of detection results from the previous stage. Then, a glimpse-based decoder is introduced to provide refined detection results based on both the glimpse features and the attention modeling outputs of the previous stage. In practice, REGO can be easily embedded in representative DETR variants while maintaining their fully end-to-end training and inference pipelines. In particular, REGO helps Deformable DETR achieve 44.8 AP on the MSCOCO dataset with only 36 training epochs, compared with the first DETR and the Deformable DETR that require 500 and 50 epochs to achieve comparable performance, respectively. Experiments also show that REGO consistently boosts the performance of different DETR detectors by up to 7% relative gain at the same setting of 50 training epochs. Code is available via https://github.com/zhechen/Deformable-DETR-REGO.