Switchable Online Knowledge Distillation

TL;DR

SwitOKD introduces an adaptive switching strategy during training to optimize the gap between teacher and student in online knowledge distillation, enhancing student performance while maintaining teacher quality.

Contribution

The paper proposes a novel switchable mechanism with an adaptive threshold to dynamically calibrate the distillation gap during training in online knowledge distillation.

Findings

Improves student accuracy over state-of-the-art methods.

Maintains teacher performance comparable to existing online distillation techniques.

Extends effectively to multiple network topologies.

Abstract

Online Knowledge Distillation (OKD) improves the involved models by reciprocally exploiting the difference between teacher and student. Several crucial bottlenecks over the gap between them -- e.g., Why and when does a large gap harm the performance, especially for student? How to quantify the gap between teacher and student? -- have received limited formal study. In this paper, we propose Switchable Online Knowledge Distillation (SwitOKD), to answer these questions. Instead of focusing on the accuracy gap at test phase by the existing arts, the core idea of SwitOKD is to adaptively calibrate the gap at training phase, namely distillation gap, via a switching strategy between two modes -- expert mode (pause the teacher while keep the student learning) and learning mode (restart the teacher). To possess an appropriate distillation gap, we further devise an adaptive switching threshold, which provides a formal criterion as to when to switch to learning mode or expert mode, and thus improves the student's performance. Meanwhile, the teacher benefits from our adaptive switching threshold and keeps basically on a par with other online arts. We further extend SwitOKD to multiple networks with two basis topologies. Finally, extensive experiments and analysis validate the merits of SwitOKD for classification over the state-of-the-arts. Our code is available at https://github.com/hfutqian/SwitOKD.