ARC: Argument Representation and Coverage Analysis for Zero-Shot Long Document Summarization with Instruction Following LLMs

TL;DR

This paper introduces ARC, a framework for evaluating how well summaries preserve key arguments in high-stakes domains, revealing models' strengths and weaknesses in argument coverage.

Contribution

ARC provides an interpretable, argument-focused evaluation method for long document summarization, highlighting systematic biases and guiding improvements in model performance.

Findings

Models capture some salient argument roles but often omit critical information.

Argument coverage is affected by context window bias and role-specific preferences.

ARC uncovers patterns that inform strategies for more reliable summarization.

Abstract

We introduce Argument Representation Coverage (ARC), a bottom-up evaluation framework that assesses how well summaries preserve salient arguments, a crucial issue in summarizing high-stakes domains such as law. ARC provides an interpretable lens by distinguishing between different information types to be covered and by separating omissions from factual errors. Using ARC, we evaluate summaries from eight open-weight large language models in two domains where argument roles are central: long legal opinions and scientific articles. Our results show that while these models capture some salient roles, they frequently omit critical information, particularly when arguments are sparsely distributed across the input. Moreover, ARC uncovers systematic patterns, showing how context window positional bias and role-specific preferences shape argument coverage, and provides actionable guidance for developing more complete and reliable summarization strategies.