A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents

TL;DR

This paper systematically reviews evaluation practices for co-speech gesture generation in embodied conversational agents, highlighting the need for standardized, rigorous assessment tools and proposing a checklist for consistent reporting.

Contribution

It provides a comprehensive review of existing evaluation methods, identifies gaps, and offers recommendations and a checklist to improve consistency and rigor in gesture evaluation studies.

Findings

Most studies use subjective, within-subject evaluation methods

Lack of standardized evaluation tools across studies

Recommendations for systematic testing and reporting of gesture generation models

Abstract

Embodied conversational agents (ECA) are often designed to produce nonverbal behavior to complement or enhance their verbal communication. One such form of nonverbal behavior is co-speech gesturing, which involves movements that the agent makes with its arms and hands that are paired with verbal communication. Co-speech gestures for ECAs can be created using different generation methods, divided into rule-based and data-driven processes, with the latter gaining traction because of the increasing interest from the applied machine learning community. However, reports on gesture generation methods use a variety of evaluation measures, which hinders comparison. To address this, we present a systematic review on co-speech gesture generation methods for iconic, metaphoric, deictic, and beat gestures, including reported evaluation methods. We review 22 studies that have an ECA with a human-like upper body that uses co-speech gesturing in social human-agent interaction. This includes studies that use human participants to evaluate performance. We found most studies use a within-subject design and rely on a form of subjective evaluation, but without a systematic approach. We argue that the field requires more rigorous and uniform tools for co-speech gesture evaluation, and formulate recommendations for empirical evaluation, including standardized phrases and example scenarios to help systematically test generative models across studies. Furthermore, we also propose a checklist that can be used to report relevant information for the evaluation of generative models, as well as to evaluate co-speech gesture use.