Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation

TL;DR

Referee introduces a reference-free, controllable sentence summarization framework using iterative Symbolic Knowledge Distillation, resulting in smaller, high-quality models with precise compression control.

Contribution

First demonstration of reference-free, controllable sentence summarization via iterative Symbolic Knowledge Distillation with explicit filters and knowledge refinement.

Findings

Models outperform GPT3-Instruct in controllability and quality

Iterative distillation produces smaller, sharper models

High-quality dataset of varied compression ratios generated

Abstract

We present Referee, a novel framework for sentence summarization that can be trained reference-free (i.e., requiring no gold summaries for supervision), while allowing direct control for compression ratio. Our work is the first to demonstrate that reference-free, controlled sentence summarization is feasible via the conceptual framework of Symbolic Knowledge Distillation (West et al., 2022), where latent knowledge in pre-trained language models is distilled via explicit examples sampled from the teacher models, further purified with three types of filters: length, fidelity, and Information Bottleneck. Moreover, we uniquely propose iterative distillation of knowledge, where student models from the previous iteration of distillation serve as teacher models in the next iteration. Starting off from a relatively modest set of GPT3-generated summaries, we demonstrate how iterative knowledge distillation can lead to considerably smaller, but better summarizers with sharper controllability. A useful by-product of this iterative distillation process is a high-quality dataset of sentence-summary pairs with varying degrees of compression ratios. Empirical results demonstrate that the final student models vastly outperform the much larger GPT3-Instruct model in terms of the controllability of compression ratios, without compromising the quality of resulting summarization.