Using Virtual Humans to Understand Real Ones

TL;DR

This study explores how virtual humans can be used to understand real human non-verbal communication by manipulating gaze, facial expression, and gestures, revealing complex interactions affecting perceived pleasure and dominance.

Contribution

It introduces a novel experimental setup using virtual agents to systematically manipulate non-verbal cues and study their effects on human affective responses.

Findings

Gaze, gesture, and facial expression interactively influence pleasure perception.

Gesture alone significantly affects perceived dominance.

Complex interactions between non-verbal cues highlight nuanced social signaling.

Abstract

Human interactions are characterized by explicit as well as implicit channels of communication. While the explicit channel transmits overt messages, the implicit ones transmit hidden messages about the communicator (e.g., his/her intentions and attitudes). There is a growing consensus that providing a computer with the ability to manipulate implicit affective cues should allow for a more meaningful and natural way of studying particular non-verbal signals of human-human communications by human-computer interactions. In this pilot study, we created a non-dynamic human-computer interaction while manipulating three specific non-verbal channels of communication: gaze pattern, facial expression, and gesture. Participants rated the virtual agent on affective dimensional scales (pleasure, arousal, and dominance) while their physiological signal (electrodermal activity, EDA) was captured during the interaction. Assessment of the behavioral data revealed a significant and complex three-way interaction between gaze, gesture, and facial configuration on the dimension of pleasure, as well as a main effect of gesture on the dimension of dominance. These results suggest a complex relationship between different non-verbal cues and the social context in which they are interpreted. Qualifying considerations as well as possible next steps are further discussed in light of these exploratory findings.