LLMs as Assessors: Right for the Right Reason?

TL;DR

This study evaluates the effectiveness of Large Language Models as relevance assessors in Information Retrieval, emphasizing their ability to highlight relevant passages and the importance of human oversight.

Contribution

It extends prior research by using a Wikipedia-based test collection to assess LLMs' judgment accuracy and reasoning in relevance evaluation.

Findings

LLMs can identify relevant passages similar to human assessors.

LLMs often make correct relevance judgments but are not yet reliable enough to replace humans.

Caution is advised in using LLMs for dataset creation due to potential for incorrect judgments.

Abstract

A good deal of recent research has focused on how Large Language Models (LLMs) may be used as judges in place of humans to evaluate the quality of the output produced by various text / image processing systems. Within this broader context, a number of studies have investigated the specific question of how effectively LLMs can be used as relevance assessors for the standard ad hoc task in Information Retrieval (IR). We extend these studies by looking at additional questions. Most importantly, we use a Wikipedia based test collection created by the INEX initiative, and prompt LLMs to not only judge whether documents are relevant / non-relevant, but to highlight relevant passages in documents that it regards as useful. The human relevance assessors involved in creating this collection were given analogous instructions, i.e., they were asked to highlight all passages within a document that respond to the information need expressed in a query. This enables us to evaluate the quality of LLMs as judges not only at the document level, but to also quantify how often these judges are right for the right reasons. Our observations lead us to reiterate the cautionary note sounded in some earlier studies when it comes to using LLMs as assessors for creating IR datasets: while LLMs are unquestionably promising, and may be used judiciously to subtantially reduce the amount of human involvement required to generate high-quality benchmark datasets, they cannot replace humans as assessors.