The Effect of Document Summarization on LLM-Based Relevance Judgments

TL;DR

This study explores how text summarization influences the reliability of LLM-based relevance judgments in IR evaluation, finding that summaries can maintain ranking stability but also introduce biases affecting judgment accuracy.

Contribution

It systematically evaluates the impact of document summarization on LLM-based relevance assessments across multiple datasets, highlighting implications for IR evaluation reliability.

Findings

Summary-based judgments maintain system ranking stability.

Summarization introduces biases and shifts in label distributions.

Summarization offers a more efficient approach for large-scale IR evaluation.

Abstract

Relevance judgments are central to the evaluation of Information Retrieval (IR) systems, but obtaining them from human annotators is costly and time-consuming. Large Language Models (LLMs) have recently been proposed as automated assessors, showing promising alignment with human annotations. Most prior studies have treated documents as fixed units, feeding their full content directly to LLM assessors. We investigate how text summarization affects the reliability of LLM-based judgments and their downstream impact on IR evaluation. Using state-of-the-art LLMs across multiple TREC collections, we compare judgments made from full documents with those based on LLM-generated summaries of different lengths. We examine their agreement with human labels, their effect on retrieval effectiveness evaluation, and their influence on IR systems' ranking stability. Our findings show that summary-based judgments achieve comparable stability in systems' ranking to full-document judgments, while introducing systematic shifts in label distributions and biases that vary by model and dataset. These results highlight summarization as both an opportunity for more efficient large-scale IR evaluation and a methodological choice with important implications for the reliability of automatic judgments.