A Workbench for Autograding Retrieve/Generate Systems

TL;DR

This paper introduces a workbench for evaluating autoregressive LLM-based IR systems using alternative methods like relevance judgments, key fact coverage, and exam question answering, addressing limitations of traditional passage-level assessments.

Contribution

It presents a novel evaluation workbench that leverages LLMs for assessing IR system responses through multiple innovative approaches.

Findings

LLMs can effectively judge response relevance.

The workbench enables development of new test collections.

Evaluation methods impact system ranking and development.

Abstract

This resource paper addresses the challenge of evaluating Information Retrieval (IR) systems in the era of autoregressive Large Language Models (LLMs). Traditional methods relying on passage-level judgments are no longer effective due to the diversity of responses generated by LLM-based systems. We provide a workbench to explore several alternative evaluation approaches to judge the relevance of a system's response that incorporate LLMs: 1. Asking an LLM whether the response is relevant; 2. Asking the LLM which set of nuggets (i.e., relevant key facts) is covered in the response; 3. Asking the LLM to answer a set of exam questions with the response. This workbench aims to facilitate the development of new, reusable test collections. Researchers can manually refine sets of nuggets and exam questions, observing their impact on system evaluation and leaderboard rankings. Resource available at https://github.com/TREMA-UNH/autograding-workbench