Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization

TL;DR

This paper introduces MaKD, a multi-aspect knowledge distillation method that mimics self-attention and feed-forward modules to better preserve fine-grained language knowledge during model compression.

Contribution

MaKD extends knowledge distillation by capturing multiple aspects of language models, improving performance over existing layer-focused methods.

Findings

MaKD achieves competitive results with strong baselines.

MaKD performs well in distilling auto-regressive models.

The method captures rich language knowledge at different aspects.

Abstract

Knowledge distillation is an effective technique for pre-trained language model compression. However, existing methods only focus on the knowledge distribution among layers, which may cause the loss of fine-grained information in the alignment process. To address this issue, we introduce the Multi-aspect Knowledge Distillation (MaKD) method, which mimics the self-attention and feed-forward modules in greater depth to capture rich language knowledge information at different aspects. Experimental results demonstrate that MaKD can achieve competitive performance compared with various strong baselines with the same storage parameter budget. In addition, our method also performs well in distilling auto-regressive architecture models.