Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing

TL;DR

This paper systematically studies the impact of various components in knowledge distillation for NLP, introduces a meta framework called Distiller, and identifies key factors influencing performance across datasets.

Contribution

It proposes Distiller, a comprehensive meta framework for analyzing and optimizing knowledge distillation components in NLP, including a universal MI objective and an AutoDistiller algorithm.

Findings

Intermediate representation distillation is most critical for KD performance.

MI-$ extalpha$ achieves superior results among MI objectives.

Data augmentation significantly benefits small datasets and models.

Abstract

We aim to identify how different components in the KD pipeline affect the resulting performance and how much the optimal KD pipeline varies across different datasets/tasks, such as the data augmentation policy, the loss function, and the intermediate representation for transferring the knowledge between teacher and student. To tease apart their effects, we propose Distiller, a meta KD framework that systematically combines a broad range of techniques across different stages of the KD pipeline, which enables us to quantify each component's contribution. Within Distiller, we unify commonly used objectives for distillation of intermediate representations under a universal mutual information (MI) objective and propose a class of MI-$\alpha$ objective functions with better bias/variance trade-off for estimating the MI between the teacher and the student. On a diverse set of NLP datasets, the best Distiller configurations are identified via large-scale hyperparameter optimization. Our experiments reveal the following: 1) the approach used to distill the intermediate representations is the most important factor in KD performance, 2) among different objectives for intermediate distillation, MI-$\alpha$ performs the best, and 3) data augmentation provides a large boost for small training datasets or small student networks. Moreover, we find that different datasets/tasks prefer different KD algorithms, and thus propose a simple AutoDistiller algorithm that can recommend a good KD pipeline for a new dataset.