OD-DETR: Online Distillation for Stabilizing Training of Detection Transformer

TL;DR

OD-DETR introduces an online distillation method that stabilizes training and enhances performance of DETR models by leveraging a teacher model's knowledge without increasing parameters.

Contribution

The paper proposes a novel online distillation approach that improves DETR training stability and accuracy by utilizing teacher-guided query matching and multi-stage query distillation.

Findings

Training stability of DETR is significantly improved.

Model performance increases without additional parameters.

Training convergence is accelerated with the proposed method.

Abstract

DEtection TRansformer (DETR) becomes a dominant paradigm, mainly due to its common architecture with high accuracy and no post-processing. However, DETR suffers from unstable training dynamics. It consumes more data and epochs to converge compared with CNN-based detectors. This paper aims to stabilize DETR training through the online distillation. It utilizes a teacher model, accumulated by Exponential Moving Average (EMA), and distills its knowledge into the online model in following three aspects. First, the matching relation between object queries and ground truth (GT) boxes in the teacher is employed to guide the student, so queries within the student are not only assigned labels based on their own predictions, but also refer to the matching results from the teacher. Second, the teacher's initial query is given to the online student, and its prediction is directly constrained by the corresponding output from the teacher. Finally, the object queries from teacher's different decoding stages are used to build the auxiliary groups to accelerate the convergence. For each GT, two queries with the least matching costs are selected into this extra group, and they predict the GT box and participate the optimization. Extensive experiments show that the proposed OD-DETR successfully stabilizes the training, and significantly increases the performance without bringing in more parameters.