Exploring Iterative Controllable Summarization with Large Language Models

TL;DR

This paper investigates the controllability of large language models in abstractive summarization, introduces new evaluation metrics, and proposes a guide-to-explain framework that improves attribute alignment with fewer iterations.

Contribution

It systematically evaluates LLM controllability, introduces iterative metrics, and presents the GTE framework for more accurate and efficient controllable summarization.

Findings

LLMs struggle more with numerical attributes than linguistic ones.

GTE framework reduces the number of iterations needed for attribute alignment.

The proposed method improves controllability and summary quality.

Abstract

Large language models (LLMs) have demonstrated remarkable performance in abstractive summarization tasks. However, their ability to precisely control summary attributes (e.g., length or topic) remains underexplored, limiting their adaptability to specific user preferences. In this paper, we systematically explore the controllability of LLMs. To this end, we revisit summary attribute measurements and introduce iterative evaluation metrics, failure rate and average iteration count to precisely evaluate controllability of LLMs, rather than merely assessing errors. Our findings show that LLMs struggle more with numerical attributes than with linguistic attributes. To address this challenge, we propose a guide-to-explain framework (GTE) for controllable summarization. Our GTE framework enables the model to identify misaligned attributes in the initial draft and guides it in self-explaining errors in the previous output. By allowing the model to reflect on its misalignment, GTE generates well-adjusted summaries that satisfy the desired attributes with robust effectiveness, requiring surprisingly fewer iterations than other iterative approaches.