Accelerating DETR Convergence via Semantic-Aligned Matching

TL;DR

SAM-DETR significantly accelerates DETR's convergence by aligning semantic spaces and focusing on salient points, reducing training time while maintaining high detection accuracy.

Contribution

Introduces a semantic-aligned matching approach for DETR that improves convergence speed with minimal additional computational cost.

Findings

Achieves faster convergence compared to baseline DETR.

Maintains competitive object detection accuracy.

Requires only slight computational overhead.

Abstract

The recently developed DEtection TRansformer (DETR) establishes a new object detection paradigm by eliminating a series of hand-crafted components. However, DETR suffers from extremely slow convergence, which increases the training cost significantly. We observe that the slow convergence is largely attributed to the complication in matching object queries with target features in different feature embedding spaces. This paper presents SAM-DETR, a Semantic-Aligned-Matching DETR that greatly accelerates DETR's convergence without sacrificing its accuracy. SAM-DETR addresses the convergence issue from two perspectives. First, it projects object queries into the same embedding space as encoded image features, where the matching can be accomplished efficiently with aligned semantics. Second, it explicitly searches salient points with the most discriminative features for semantic-aligned matching, which further speeds up the convergence and boosts detection accuracy as well. Being like a plug and play, SAM-DETR complements existing convergence solutions well yet only introduces slight computational overhead. Extensive experiments show that the proposed SAM-DETR achieves superior convergence as well as competitive detection accuracy. The implementation codes are available at https://github.com/ZhangGongjie/SAM-DETR.