A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training

TL;DR

This paper systematically investigates knowledge distillation techniques for compressing natural language generation models, emphasizing task-specific training, pseudo-target augmentation, and validation with minimal labeled data, to improve efficiency in real-world applications.

Contribution

It introduces a family of pseudo-target augmentation methods and the joint-teaching approach, advancing task-specific knowledge distillation for NLG models, especially under limited labeled data conditions.

Findings

Pseudo-target augmentation significantly improves distillation.

Joint-teaching enhances knowledge transfer from teacher to student.

Effective model compression achieved with minimal labeled data.

Abstract

Modern Natural Language Generation (NLG) models come with massive computational and storage requirements. In this work, we study the potential of compressing them, which is crucial for real-world applications serving millions of users. We focus on Knowledge Distillation (KD) techniques, in which a small student model learns to imitate a large teacher model, allowing to transfer knowledge from the teacher to the student. In contrast to much of the previous work, our goal is to optimize the model for a specific NLG task and a specific dataset. Typically in real-world applications, in addition to labeled data there is abundant unlabeled task-specific data, which is crucial for attaining high compression rates via KD. In this work, we conduct a systematic study of task-specific KD techniques for various NLG tasks under realistic assumptions. We discuss the special characteristics of NLG distillation and particularly the exposure bias problem. Following, we derive a family of Pseudo-Target (PT) augmentation methods, substantially extending prior work on sequence-level KD. We propose the Joint-Teaching method, which applies word-level KD to multiple PTs generated by both the teacher and the student. Finally, we validate our findings in an extreme setup with no labeled examples using GPT-4 as the teacher. Our study provides practical model design observations and demonstrates the effectiveness of PT training for task-specific KD in NLG.