Query-Document Dense Vectors for LLM Relevance Judgment Bias Analysis

TL;DR

This paper introduces a clustering-based framework to analyze systematic biases in LLM relevance judgments, revealing specific semantic areas where LLMs diverge from human assessments, thus improving IR evaluation reliability.

Contribution

It proposes a novel representational and clustering method to localize and understand systematic biases in LLM relevance judgments compared to humans.

Findings

Systematic disagreements are concentrated in specific semantic clusters.

LLMs often under-recall relevant content in definition and policy-related queries.

Disagreement hotspots are identified where LLMs show consistent biases.

Abstract

Large Language Models (LLMs) have been used as relevance assessors for Information Retrieval (IR) evaluation collection creation due to reduced cost and increased scalability as compared to human assessors. While previous research has looked at the reliability of LLMs as compared to human assessors, in this work, we aim to understand if LLMs make systematic mistakes when judging relevance, rather than just understanding how good they are on average. To this aim, we propose a novel representational method for queries and documents that allows us to analyze relevance label distributions and compare LLM and human labels to identify patterns of disagreement and localize systematic areas of disagreement. We introduce a clustering-based framework that embeds query-document (Q-D) pairs into a joint semantic space, treating relevance as a relational property. Experiments on TREC Deep Learning 2019 and 2020 show that systematic disagreement between humans and LLMs is concentrated in specific semantic clusters rather than distributed randomly. Query-level analyses reveal recurring failures, most often in definition-seeking, policy-related, or ambiguous contexts. Queries with large variation in agreement across their clusters emerge as disagreement hotspots, where LLMs tend to under-recall relevant content or over-include irrelevant material. This framework links global diagnostics with localized clustering to uncover hidden weaknesses in LLM judgments, enabling bias-aware and more reliable IR evaluation.