InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT

TL;DR

InheritSumm is a distilled, compact summarization model derived from GPT-3.5 that achieves comparable or superior performance to GPT-3.5 in zero-shot, few-shot, and fine-tuning settings, offering a cost-effective alternative.

Contribution

The paper introduces InheritSumm, a versatile summarizer distilled from GPT-3.5, combining high performance with compactness for practical applications.

Findings

InheritSumm matches GPT-3.5 in zero-shot and few-shot summarization.

It outperforms previous small models in fine-tuning scenarios.

InheritSumm is suitable for both prefix-tuning and full-data fine-tuning.

Abstract

While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications. Conversely, previous studies have found that although automatic metrics tend to favor smaller fine-tuned models, the quality of the summaries they generate is inferior to that of larger models like GPT-3 when assessed by human evaluators. To address this issue, we propose InheritSumm, a versatile and compact summarization model derived from GPT-3.5 through distillation. InheritSumm not only exhibits comparable zeroshot and fewshot summarization capabilities to GPT-3.5 but is also sufficiently compact for fine-tuning purposes. Experimental results demonstrate that InheritSumm achieves similar or superior performance to GPT-3.5 in zeroshot and fewshot settings. Furthermore, it outperforms the previously established best small models in both prefix-tuning and full-data fine-tuning scenarios.