Input Order Shapes LLM Semantic Alignment in Multi-Document Summarization

TL;DR

This study shows that large language models tend to prioritize the first input document when generating summaries, which can bias the output and impact downstream applications.

Contribution

It reveals a primacy effect in LLM summarization, demonstrating that input order significantly influences semantic alignment in multi-document summarization.

Findings

Significant primacy effect observed for BERTScore.

Summaries are more aligned with the first document.

Later documents have less influence on the summary.

Abstract

Large language models (LLMs) are now used in settings such as Google's AI Overviews, where it summarizes multiple long documents. However, it remains unclear whether they weight all inputs equally. Focusing on abortion-related news, we construct 40 pro-neutral-con article triplets, permute each triplet into six input orders, and prompt Gemini 2.5 Flash to generate a neutral overview. We evaluate each summary against its source articles using ROUGE-L (lexical overlap), BERTScore (semantic similarity), and SummaC (factual consistency). One-way ANOVA reveals a significant primacy effect for BERTScore across all stances, indicating that summaries are more semantically aligned with the first-seen article. Pairwise comparisons further show that Position 1 differs significantly from Positions 2 and 3, while the latter two do not differ from each other, confirming a selective preference for the first document. The findings present risks for applications that rely on LLM-generated overviews and for agentic AI systems, where the steps involving LLMs can disproportionately influence downstream actions.