Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

TL;DR

This paper develops a theory explaining how ensembles of neural networks improve test accuracy through multi-view data structures and how this knowledge can be distilled into single models, revealing new insights into deep learning ensemble methods.

Contribution

It introduces a novel theoretical framework showing ensemble and knowledge distillation effectiveness in deep learning differ from traditional theories, especially under multi-view data structures.

Findings

Ensemble of neural networks can provably improve test accuracy with multi-view data.

Knowledge distillation can transfer ensemble performance into a single model.

Self-distillation combines ensemble and knowledge distillation effects to enhance accuracy.

Abstract

We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (such as boosting or NTKs, neural tangent kernels). To properly understand them, we develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.