Towards Objective Evaluation of Socially-Situated Conversational Robots: Assessing Human-Likeness through Multimodal User Behaviors

TL;DR

This paper introduces an objective, multimodal behavior-based evaluation method for socially situated conversational robots, focusing on assessing human-likeness through observable user behaviors to improve reproducibility.

Contribution

It presents a novel approach that uses multimodal user behaviors to objectively evaluate the human-likeness of conversational robots, moving beyond subjective assessments.

Findings

High correlation between user behaviors and human-likeness scores

Created an annotated dataset for behavior-based evaluation

Demonstrated feasibility of behavior-based assessment method

Abstract

This paper tackles the challenging task of evaluating socially situated conversational robots and presents a novel objective evaluation approach that relies on multimodal user behaviors. In this study, our main focus is on assessing the human-likeness of the robot as the primary evaluation metric. While previous research often relied on subjective evaluations from users, our approach aims to evaluate the robot's human-likeness based on observable user behaviors indirectly, thus enhancing objectivity and reproducibility. To begin, we created an annotated dataset of human-likeness scores, utilizing user behaviors found in an attentive listening dialogue corpus. We then conducted an analysis to determine the correlation between multimodal user behaviors and human-likeness scores, demonstrating the feasibility of our proposed behavior-based evaluation method.