Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

TL;DR

This survey comprehensively reviews knowledge distillation techniques for Large Language Models, focusing on methods, evaluation, and applications to address challenges of size and computational demands.

Contribution

It categorizes distillation methods into white-box and black-box approaches, analyzes evaluation metrics, and suggests future research directions for LLM compression.

Findings

White-box and black-box distillation methods differ significantly.

Distillation improves inference speed with minimal performance loss.

Future research should focus on evaluation benchmarks and practical applications.

Abstract

Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This paper presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods, and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field.