Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation

TL;DR

This paper introduces Spaced KD, a knowledge distillation strategy inspired by the biological spacing effect, which improves neural network generalization by training with spaced intervals, leading to flatter loss landscapes and better performance.

Contribution

It proposes a novel Spaced KD method that enhances online and self knowledge distillation by incorporating spaced training intervals, inspired by biological learning theories.

Findings

Spaced KD improves DNN performance by up to 2.31% on Tiny-ImageNet.

It leads to convergence on flatter loss landscapes during training.

Experimental results validate the effectiveness of the proposed strategy.

Abstract

Knowledge distillation (KD) is a powerful strategy for training deep neural networks (DNNs). Although it was originally proposed to train a more compact "student" model from a large "teacher" model, many recent efforts have focused on adapting it to promote generalization of the model itself, such as online KD and self KD. Here, we propose an accessible and compatible strategy named Spaced KD to improve the effectiveness of both online KD and self KD, in which the student model distills knowledge from a teacher model trained with a space interval ahead. This strategy is inspired by a prominent theory named spacing effect in biological learning and memory, positing that appropriate intervals between learning trials can significantly enhance learning performance. With both theoretical and empirical analyses, we demonstrate that the benefits of the proposed Spaced KD stem from convergence to a flatter loss landscape during stochastic gradient descent (SGD). We perform extensive experiments to validate the effectiveness of Spaced KD in improving the learning performance of DNNs (e.g., the performance gain is up to 2.31% and 3.34% on Tiny-ImageNet over online KD and self KD, respectively). Our codes have been released on github https://github.com/SunGL001/Spaced-KD.