When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search

TL;DR

This paper investigates how disagreement between large language models in relevance labeling affects retrieval, revealing systematic biases and proposing disagreement analysis as a tool for evaluation in policy-relevant searches.

Contribution

It demonstrates that LLM disagreement is systematic, not random, and introduces classification disagreement analysis as a new approach for evaluating retrieval systems.

Findings

Disagreement cases show consistent lexical patterns.

Disagreement leads to divergent top-ranked outputs.

Simple classifiers can predict disagreement with high AUCs above 0.74.

Abstract

Large language models (LLMs) are increasingly used to assign document relevance labels in information retrieval pipelines, especially in domains lacking human-labeled data. However, different models often disagree on borderline cases, raising concerns about how such disagreement affects downstream retrieval. This study examines labeling disagreement between two open-weight LLMs, LLaMA and Qwen, on a corpus of scholarly abstracts related to Sustainable Development Goals (SDGs) 1, 3, and 7. We isolate disagreement subsets and examine their lexical properties, rank-order behavior, and classification predictability. Our results show that model disagreement is systematic, not random: disagreement cases exhibit consistent lexical patterns, produce divergent top-ranked outputs under shared scoring functions, and are distinguishable with AUCs above 0.74 using simple classifiers. These findings suggest that LLM-based filtering introduces structured variability in document retrieval, even under controlled prompting and shared ranking logic. We propose using classification disagreement as an object of analysis in retrieval evaluation, particularly in policy-relevant or thematic search tasks.