Evaluating Open-Domain Question Answering in the Era of Large Language Models

TL;DR

This paper critically examines the limitations of lexical matching for open-domain QA evaluation, highlighting the improved performance of LLMs like InstructGPT and proposing alternative evaluation methods including regex and automated models, with human judgment remaining essential.

Contribution

It provides a comprehensive analysis of QA evaluation challenges, demonstrates the inadequacy of lexical matching for LLM answers, and evaluates alternative automated and regex-based methods.

Findings

InstructGPT (zero-shot) improves performance by nearly 60%.

Over 50% of lexical failures are due to semantically equivalent answers.

Regex matching aligns better with human judgments than lexical matching.

Abstract

Lexical matching remains the de facto evaluation method for open-domain question answering (QA). Unfortunately, lexical matching fails completely when a plausible candidate answer does not appear in the list of gold answers, which is increasingly the case as we shift from extractive to generative models. The recent success of large language models (LLMs) for QA aggravates lexical matching failures since candidate answers become longer, thereby making matching with the gold answers even more challenging. Without accurate evaluation, the true progress in open-domain QA remains unknown. In this paper, we conduct a thorough analysis of various open-domain QA models, including LLMs, by manually evaluating their answers on a subset of NQ-open, a popular benchmark. Our assessments reveal that while the true performance of all models is significantly underestimated, the performance of the InstructGPT (zero-shot) LLM increases by nearly +60%, making it on par with existing top models, and the InstructGPT (few-shot) model actually achieves a new state-of-the-art on NQ-open. We also find that more than 50% of lexical matching failures are attributed to semantically equivalent answers. We further demonstrate that regex matching ranks QA models consistent with human judgments, although still suffering from unnecessary strictness. Finally, we demonstrate that automated evaluation models are a reasonable surrogate for lexical matching in some circumstances, but not for long-form answers generated by LLMs. The automated models struggle in detecting hallucinations in LLM answers and are thus unable to evaluate LLMs. At this time, there appears to be no substitute for human evaluation.