Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs

TL;DR

Soda-Eval introduces a large, GPT-4-annotated dataset for evaluating open-domain dialogue models, revealing ongoing challenges and improvements in automated dialogue assessment.

Contribution

The paper presents Soda-Eval, a new extensive dataset for dialogue evaluation, and analyzes the performance of instruction-tuned LLMs using this benchmark.

Findings

Current chatbots show issues with coherence and commonsense.

Fine-tuning LLMs improves evaluation performance.

Automated evaluation remains a challenging task.

Abstract

Although human evaluation remains the gold standard for open-domain dialogue evaluation, the growing popularity of automated evaluation using Large Language Models (LLMs) has also extended to dialogue. However, most frameworks leverage benchmarks that assess older chatbots on aspects such as fluency and relevance, which are not reflective of the challenges associated with contemporary models. In fact, a qualitative analysis on Soda, a GPT-3.5 generated dialogue dataset, suggests that current chatbots may exhibit several recurring issues related to coherence and commonsense knowledge, but generally produce highly fluent and relevant responses. Noting the aforementioned limitations, this paper introduces Soda-Eval, an annotated dataset based on Soda that covers over 120K turn-level assessments across 10K dialogues, where the annotations were generated by GPT-4. Using Soda-Eval as a benchmark, we then study the performance of several open-access instruction-tuned LLMs, finding that dialogue evaluation remains challenging. Fine-tuning these models improves performance over few-shot inferences, both in terms of correlation and explanation.