Controllable Multi-document Summarization: Coverage & Coherence Intuitive Policy with Large Language Model Based Rewards

TL;DR

This paper proposes a controllable multi-document summarization method that uses a novel coverage and coherence policy, leveraging large language models to refine extracted content, resulting in improved coherence and competitive ROUGE scores.

Contribution

It introduces a new controllable content extraction scheme with an intuitive policy for coverage and coherence, guided by LLM-based rewards, enhancing multi-document summarization.

Findings

Achieves competitive ROUGE scores in evaluation.

Outperforms baselines in coherence based on human judgment.

Demonstrates effective controllability in summarization tasks.

Abstract

Memory-efficient large language models are good at refining text input for better readability. However, controllability is a matter of concern when it comes to text generation tasks with long inputs, such as multi-document summarization. In this work, we investigate for a generic controllable approach for multi-document summarization that leverages the capabilities of LLMs to refine the text. In particular, we train a controllable content extraction scheme to extract the text that will be refined by an LLM. The scheme is designed with a novel coverage and coherence intuitive policy, which is duly rewarded by a passively trained LLM. Our approach yields competitive results in the evaluation using ROUGE metrics and outperforms potential baselines in coherence, as per human evaluation.