Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing

TL;DR

This paper introduces Multi2, a scalable framework for multi-document summarization using test-time prompt ensemble and novel LLM-based metrics, improving summary quality and understanding scaling limits.

Contribution

It presents a new test-time scaling approach for MDS with prompt ensembles and introduces two metrics for better evaluation of summaries.

Findings

Enhanced summary quality through prompt ensemble methods.

New metrics (CAP and LLM-ACU) effectively evaluate summary consistency.

Identified practical scaling boundaries for multi-document summarization.

Abstract

Recent advances in test-time scaling have shown promising results in improving Large Language Model (LLM) performance through strategic computation allocation during inference. While this approach has demonstrated strong improvements in logical and mathematical reasoning tasks, its application to natural language generation (NLG), particularly summarization, remains unexplored. Multi-Document Summarization (MDS), a fundamental task in NLG, presents unique challenges by requiring models to extract and synthesize essential information across multiple lengthy documents. Unlike reasoning tasks, MDS demands a more nuanced approach to prompt design and ensemble methods, as no single "best" prompt can satisfy diverse summarization requirements. We propose a novel framework leveraging test-time scaling for MDS. Our approach employs prompt ensemble techniques to generate multiple candidate summaries using various prompts, then combines them with an aggregator to produce a refined summary. To evaluate our method effectively, we also introduce two new LLM-based metrics: the Consistency-Aware Preference (CAP) score and LLM Atom-Content-Unit (LLM-ACU) score, which assess summary quality while addressing the positional bias inherent in traditional automatic evaluation. Our extensive experiments demonstrate that this framework significantly enhances summary quality while also revealing the practical scaling boundaries to MDS tasks.