Error-Aware Knowledge Distillation via Targeted Revision for Customer-Service Summarization

TL;DR

This paper presents a novel ARF pipeline that improves open-source language models' customer service summarization by analyzing errors, targeted revision, and fine-tuning, surpassing larger proprietary models in performance.

Contribution

The paper introduces the ARF pipeline, a new framework combining error analysis and targeted revision to significantly enhance the performance of smaller open-source LLMs.

Findings

Smaller models outperform GPT-3.5 in summarization tasks.

ARF pipeline reduces costs and improves data privacy.

Refined training data leads to superior model performance.

Abstract

We introduce an Analyze-Revise-Finetune (ARF) pipeline that enables smaller open-source language models (LLMs) to surpass substantially larger proprietary models in customer service summarization tasks. The pipeline first analyzes and categorizes common errors in summaries produced by a teacher model (GPT-3.5), then performs a targeted revision using a compact editor model (Llama 3.1 70B) to generate high-quality, refined training data. Fine-tuning smaller student models (e.g., Llama 3.1 8B, QWen3 4B) on this refined data resulted in superior summarization performance compared to GPT-3.5. The ARF pipeline improves cost efficiency and data privacy while maintaining competitive accuracy, illustrating a generalizable framework for enhancing open-source LLMs across diverse downstream applications.