Attention Temperature Matters in Abstractive Summarization Distillation

TL;DR

This paper demonstrates that adjusting attention temperatures in Transformer models enhances the effectiveness of distillation for abstractive summarization, leading to smaller models with better performance and more abstractive summaries.

Contribution

It introduces a simple yet effective method of manipulating attention temperatures to improve pseudo-labeling in Transformer-based summarization distillation.

Findings

Improved distillation performance over vanilla methods.

Generated summaries are shorter and more abstractive.

Method is effective across multiple datasets.

Abstract

Recent progress of abstractive text summarization largely relies on large pre-trained sequence-to-sequence Transformer models, which are computationally expensive. This paper aims to distill these large models into smaller ones for faster inference and minimal performance loss. Pseudo-labeling based methods are popular in sequence-to-sequence model distillation. In this paper, we find simply manipulating attention temperatures in Transformers can make pseudo labels easier to learn for student models. Our experiments on three summarization datasets show our proposed method consistently improves over vanilla pseudo-labeling based methods. We also find that both the pseudo labels and summaries produced by our students are shorter and more abstractive. Our code is available at \url{https://github.com/Shengqiang-Zhang/plate}.