Evaluating Data-Driven Co-Speech Gestures of Embodied Conversational Agents through Real-Time Interaction

TL;DR

This study introduces a novel framework for evaluating data-driven embodied conversational agents through real-time interaction, incorporating gaze tracking to assess how gestures influence human perception in live settings.

Contribution

It is the first to evaluate data-driven ECAs via real-time interaction and gaze tracking, providing new insights into gesture effects on human perception.

Findings

Gestures increased perceived human-likeness and animacy.

Real-time interaction provided more accurate perception data.

Gaze tracking revealed attention focus differences related to gestures.

Abstract

Embodied Conversational Agents that make use of co-speech gestures can enhance human-machine interactions in many ways. In recent years, data-driven gesture generation approaches for ECAs have attracted considerable research attention, and related methods have continuously improved. Real-time interaction is typically used when researchers evaluate ECA systems that generate rule-based gestures. However, when evaluating the performance of ECAs based on data-driven methods, participants are often required only to watch pre-recorded videos, which cannot provide adequate information about what a person perceives during the interaction. To address this limitation, we explored use of real-time interaction to assess data-driven gesturing ECAs. We provided a testbed framework, and investigated whether gestures could affect human perception of ECAs in the dimensions of human-likeness, animacy, perceived intelligence, and focused attention. Our user study required participants to interact with two ECAs - one with and one without hand gestures. We collected subjective data from the participants' self-report questionnaires and objective data from a gaze tracker. To our knowledge, the current study represents the first attempt to evaluate data-driven gesturing ECAs through real-time interaction and the first experiment using gaze-tracking to examine the effect of ECAs' gestures.