SEAM: A Stochastic Benchmark for Multi-Document Tasks

TL;DR

SEAM is a new benchmark designed to evaluate large language models on complex multi-document tasks, addressing challenges like contradiction and omission, and revealing insights through stochastic evaluation methods.

Contribution

The paper introduces SEAM, a stochastic benchmark for multi-document tasks, with standardized evaluation protocols and a focus on model sensitivity to prompt variations.

Findings

Multi-document tasks are challenging for state-of-the-art LLMs.

Stochastic evaluation uncovers statistical trends not visible in static benchmarks.

SEAM facilitates consistent and meaningful evaluation of multi-document capabilities.

Abstract

Various tasks, such as summarization, multi-hop question answering, or coreference resolution, are naturally phrased over collections of real-world documents. Such tasks present a unique set of challenges, revolving around the lack of coherent narrative structure across documents, which often leads to contradiction, omission, or repetition of information. Despite their real-world application and challenging properties, there is currently no benchmark which specifically measures the abilities of large language models (LLMs) on multi-document tasks. To bridge this gap, we present SEAM (a Stochastic Evaluation Approach for Multi-document tasks), a conglomerate benchmark over a diverse set of multi-document datasets, setting conventional evaluation criteria, input-output formats, and evaluation protocols. In particular, SEAM addresses the sensitivity of LLMs to minor prompt variations through repeated evaluations, where in each evaluation we sample uniformly at random the values of arbitrary factors (e.g., the order of documents). We evaluate different LLMs on SEAM finding that multi-document tasks pose a significant challenge for LLMs, even for state-of-the-art models with 70B parameters. In addition, we show that the stochastic approach uncovers underlying statistical trends which cannot be observed in a static benchmark. We hope that SEAM will spur progress via consistent and meaningful evaluation of multi-document tasks.