Evaluator for Emotionally Consistent Chatbots

TL;DR

This paper introduces an evaluator designed to assess the emotional consistency of chatbots, addressing a key gap in current evaluation methods that focus on coherence and fluency.

Contribution

It proposes a novel training approach for an evaluator specifically targeting emotional consistency in chatbot responses.

Findings

Evaluator effectively measures emotional consistency

Improves assessment accuracy over existing metrics

Enhances chatbot development with emotional awareness

Abstract

One challenge for evaluating current sequence- or dialogue-level chatbots, such as Empathetic Open-domain Conversation Models, is to determine whether the chatbot performs in an emotionally consistent way. The most recent work only evaluates on the aspects of context coherence, language fluency, response diversity, or logical self-consistency between dialogues. This work proposes training an evaluator to determine the emotional consistency of chatbots.