Evaluate What You Can't Evaluate: Unassessable Quality for Generated Response

TL;DR

This paper investigates the reliability of large language models as reference-free evaluators for dialogue responses, revealing their limitations in identifying unreasonable outputs, especially in closed-ended cases requiring external knowledge.

Contribution

The study introduces challenging adversarial datasets for evaluating LLM-based evaluators and highlights their current shortcomings in assessing dialogue response quality.

Findings

LLMs struggle to identify unreasonable responses in dialogue evaluation.

Adversarial datasets reveal limitations of LLM evaluators in closed-ended scenarios.

Risks exist in relying solely on LLMs for response quality assessment.

Abstract

LLMs (large language models) such as ChatGPT have shown remarkable language understanding and generation capabilities. Although reference-free evaluators based on LLMs show better human alignment than traditional reference-based evaluators, there are many challenges in using reference-free evaluators based on LLMs. Reference-free evaluators are more suitable for open-ended examples with different semantics responses. But not all examples are open-ended. For closed-ended examples with unique correct semantic response, reference-free evaluators will still consider it high quality when giving a response that is inconsistent with the facts and the semantic of reference. In order to comprehensively evaluate the reliability of evaluators based on LLMs, we construct two adversarial meta-evaluation dialogue generation datasets KdConv-ADV and DSTC7-ADV based on KdConv and DSTC7-AVSD, respectively. Compared to previous meta-evaluation benchmarks, KdConv-ADV and DSTC7-ADV are much more challenging since they requires evaluators to be able to reasonably evaluate closed-ended examples with the help of external knowledge or even its own knowledge. Empirical results show that the ability of LLMs to identify unreasonable responses is insufficient. There are risks in using eference-free evaluators based on LLMs to evaluate the quality of dialogue responses.