Return of EM: Entity-driven Answer Set Expansion for QA Evaluation

TL;DR

This paper introduces an entity-driven answer set expansion method using soft EM to improve QA evaluation, achieving high reliability, interpretability, and environmental benefits over traditional LLM-based methods.

Contribution

It presents a novel soft EM approach that expands gold answer sets based on entity types, enhancing evaluation accuracy and interpretability.

Findings

Outperforms traditional QA evaluation methods significantly

Achieves reliability comparable to LLM-based evaluations

Reduces environmental impact and improves interpretability

Abstract

Recently, directly using large language models (LLMs) has been shown to be the most reliable method to evaluate QA models. However, it suffers from limited interpretability, high cost, and environmental harm. To address these, we propose to use soft EM with entity-driven answer set expansion. Our approach expands the gold answer set to include diverse surface forms, based on the observation that the surface forms often follow particular patterns depending on the entity type. The experimental results show that our method outperforms traditional evaluation methods by a large margin. Moreover, the reliability of our evaluation method is comparable to that of LLM-based ones, while offering the benefits of high interpretability and reduced environmental harm.