ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues

TL;DR

This paper introduces ECoh, a multilingual, lightweight dialogue coherence evaluator trained on a new dataset, outperforming larger models like GPT-3.5-Turbo in coherence detection and explanation quality across multiple languages.

Contribution

The paper presents GenResCoh, a large multilingual dataset for coherence evaluation, and ECoh, a novel lightweight evaluator that surpasses larger models in multilingual dialogue coherence assessment.

Findings

ECoh outperforms GPT-3.5-Turbo in multilingual coherence detection.

ECoh's explanations are comparable in quality to those of the teacher model.

GenResCoh enables effective training of multilingual dialogue evaluators.

Abstract

Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence). GenResCoh is a novel LLM generated dataset comprising over 130k negative and positive responses and accompanying explanations seeded from XDailyDialog and XPersona covering English, French, German, Italian, and Chinese. Leveraging GenResCoh, we propose ECoh (Evaluation of Coherence), a family of evaluators trained to assess response coherence across multiple languages. Experimental results demonstrate that ECoh achieves multilingual detection capabilities superior to the teacher model (GPT-3.5-Turbo) on GenResCoh, despite being based on a much smaller architecture. Furthermore, the explanations provided by ECoh closely align in terms of quality with those generated by the teacher model.