MixKD: Towards Efficient Distillation of Large-scale Language Models

TL;DR

MixKD introduces a data-agnostic distillation method using mixup augmentation to improve the generalization of large language models, especially in low-resource scenarios, by encouraging the student to mimic the teacher on interpolated data.

Contribution

The paper proposes MixKD, a novel distillation framework that incorporates mixup augmentation to enhance model generalization and address data scarcity issues in large-scale language model compression.

Findings

MixKD outperforms standard knowledge distillation on GLUE benchmark.

MixKD shows significant gains in limited-data settings.

Ablation studies confirm the effectiveness of mixup in distillation.

Abstract

Large-scale language models have recently demonstrated impressive empirical performance. Nevertheless, the improved results are attained at the price of bigger models, more power consumption, and slower inference, which hinder their applicability to low-resource (both memory and computation) platforms. Knowledge distillation (KD) has been demonstrated as an effective framework for compressing such big models. However, large-scale neural network systems are prone to memorize training instances, and thus tend to make inconsistent predictions when the data distribution is altered slightly. Moreover, the student model has few opportunities to request useful information from the teacher model when there is limited task-specific data available. To address these issues, we propose MixKD, a data-agnostic distillation framework that leverages mixup, a simple yet efficient data augmentation approach, to endow the resulting model with stronger generalization ability. Concretely, in addition to the original training examples, the student model is encouraged to mimic the teacher's behavior on the linear interpolation of example pairs as well. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the generalization error and the empirical error. To verify its effectiveness, we conduct experiments on the GLUE benchmark, where MixKD consistently leads to significant gains over the standard KD training, and outperforms several competitive baselines. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.