Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale

TL;DR

This paper introduces Performance-Guided Knowledge Distillation (PGKD), a cost-effective method that distills large language models into smaller, efficient models for text classification, significantly reducing inference costs and latency.

Contribution

The paper presents a novel, performance-aware active learning framework for LLM knowledge distillation tailored for multi-class, sparsely annotated datasets, outperforming traditional methods.

Findings

PGKD models are up to 130X faster than LLMs.

PGKD reduces inference costs by up to 25X.

Outperforms traditional BERT-base and other distillation methods.

Abstract

Large Language Models (LLMs) face significant challenges at inference time due to their high computational demands. To address this, we present Performance-Guided Knowledge Distillation (PGKD), a cost-effective and high-throughput solution for production text classification applications. PGKD utilizes teacher-student Knowledge Distillation to distill the knowledge of LLMs into smaller, task-specific models. PGKD establishes an active learning routine between the student model and the LLM; the LLM continuously generates new training data leveraging hard-negative mining, student model validation performance, and early-stopping protocols to inform the data generation. By employing a cyclical, performance-aware approach tailored for highly multi-class, sparsely annotated datasets prevalent in industrial text classification, PGKD effectively addresses training challenges and outperforms traditional BERT-base models and other knowledge distillation methods on several multi-class classification datasets. Additionally, cost and latency benchmarking reveals that models fine-tuned with PGKD are up to 130X faster and 25X less expensive than LLMs for inference on the same classification task. While PGKD is showcased for text classification tasks, its versatile framework can be extended to any LLM distillation task, including language generation, making it a powerful tool for optimizing performance across a wide range of AI applications.