Can We Use Large Language Models to Fill Relevance Judgment Holes?

TL;DR

This paper investigates using Large Language Models to fill relevance judgment gaps in test collections, especially in dynamic contexts like conversational search, and evaluates their impact on ranking consistency.

Contribution

It introduces a method to extend test collections with LLM-generated relevance judgments grounded in human judgments, highlighting challenges and considerations for alignment.

Findings

LLM-based automatic judgments show lower correlation with human judgments.

Using LLMs on the entire document pool yields more consistent rankings.

The effect of LLM judgments varies depending on the model and hole size.

Abstract

Incomplete relevance judgments limit the re-usability of test collections. When new systems are compared against previous systems used to build the pool of judged documents, they often do so at a disadvantage due to the ``holes'' in test collection (i.e., pockets of un-assessed documents returned by the new system). In this paper, we take initial steps towards extending existing test collections by employing Large Language Models (LLM) to fill the holes by leveraging and grounding the method using existing human judgments. We explore this problem in the context of Conversational Search using TREC iKAT, where information needs are highly dynamic and the responses (and, the results retrieved) are much more varied (leaving bigger holes). While previous work has shown that automatic judgments from LLMs result in highly correlated rankings, we find substantially lower correlates when human plus automatic judgments are used (regardless of LLM, one/two/few shot, or fine-tuned). We further find that, depending on the LLM employed, new runs will be highly favored (or penalized), and this effect is magnified proportionally to the size of the holes. Instead, one should generate the LLM annotations on the whole document pool to achieve more consistent rankings with human-generated labels. Future work is required to prompt engineering and fine-tuning LLMs to reflect and represent the human annotations, in order to ground and align the models, such that they are more fit for purpose.