Knowledge Distillation by On-the-Fly Native Ensemble

TL;DR

This paper introduces an online knowledge distillation method called On-the-fly Native Ensemble (ONE) that trains a single multi-branch network while dynamically creating a strong teacher, improving generalization efficiently across multiple datasets.

Contribution

The paper proposes a novel one-stage online distillation approach that constructs a high-capacity teacher on-the-fly within a single network, eliminating the need for pre-trained teachers.

Findings

ONE outperforms alternative methods on multiple datasets.

It enhances generalization performance of various neural networks.

The approach maintains computational efficiency.

Abstract

Knowledge distillation is effective to train small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a highcapacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) strategy for one-stage online distillation. Specifically, ONE trains only a single multi-branch network while simultaneously establishing a strong teacher on-the- fly to enhance the learning of target network. Extensive evaluations show that ONE improves the generalisation performance a variety of deep neural networks more significantly than alternative methods on four image classification dataset: CIFAR10, CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency advantages.