Iterative Self Knowledge Distillation -- From Pothole Classification to Fine-Grained and COVID Recognition

TL;DR

This paper introduces an iterative self knowledge distillation method to enhance lightweight classifiers for pothole detection and other tasks, demonstrating improved accuracy across multiple datasets and architectures.

Contribution

The paper proposes a novel iterative self knowledge distillation approach that improves lightweight models without extra parameters, applicable to various classification tasks.

Findings

ISKD outperforms state-of-the-art self knowledge distillation methods.

Iterative training benefits both teacher and student models.

Applicable to diverse datasets including medical imaging and fine-grained classification.

Abstract

Pothole classification has become an important task for road inspection vehicles to save drivers from potential car accidents and repair bills. Given the limited computational power and fixed number of training epochs, we propose iterative self knowledge distillation (ISKD) to train lightweight pothole classifiers. Designed to improve both the teacher and student models over time in knowledge distillation, ISKD outperforms the state-of-the-art self knowledge distillation method on three pothole classification datasets across four lightweight network architectures, which supports that self knowledge distillation should be done iteratively instead of just once. The accuracy relation between the teacher and student models shows that the student model can still benefit from a moderately trained teacher model. Implying that better teacher models generally produce better student models, our results justify the design of ISKD. In addition to pothole classification, we also demonstrate the efficacy of ISKD on six additional datasets associated with generic classification, fine-grained classification, and medical imaging application, which supports that ISKD can serve as a general-purpose performance booster without the need of a given teacher model and extra trainable parameters.