Learning to Control Summaries with Score Ranking

TL;DR

This paper introduces a loss function that aligns summarization models with fine-grained evaluation scores, enabling improved summary quality and precise control over individual criteria like conciseness and completeness.

Contribution

It proposes a novel loss function that allows for dimension-specific control in summarization, balancing quality and trade-offs among multiple criteria.

Findings

Improves summary quality across multiple criteria.

Enables selective prioritization of specific summary dimensions.

Achieves performance comparable to state-of-the-art models with added controllability.

Abstract

Recent advances in summarization research focus on improving summary quality across multiple criteria, such as completeness, conciseness, and faithfulness, by jointly optimizing these dimensions. However, these efforts largely overlook the challenge of controlling summary generation with respect to individual criteria, especially in the presence of their inherent trade-offs. For example, enhancing conciseness can compromise completeness, and vice versa. In this work, we address this gap by proposing a loss function that aligns model outputs with fine-grained, model-based evaluation scores (e.g., from FineSurE), enabling both improvement in summary quality and dimension-specific control. Our approach improves the overall quality of summaries while maintaining the ability to selectively prioritize one criterion over others. Experiments on three pretrained models (LLaMA, Qwen, and Mistral) demonstrate that our method achieves performance comparable to state-of-the-art summarizers, while uniquely offering strong controllability over individual quality dimensions.