AMII: Adaptive Multimodal Inter-personal and Intra-personal Model for Adapted Behavior Synthesis

TL;DR

AMII is a novel model that synthesizes adaptive multimodal facial gestures for social agents, effectively capturing intra- and inter-personal relationships to improve interaction realism.

Contribution

It introduces a modality memory encoding schema with attention mechanisms for adaptive facial gesture synthesis in SIAs, handling role interchangeability.

Findings

Outperforms state-of-the-art methods in objective evaluations.

Effectively models intra- and inter-personal relationships.

Enhances the realism of social agent interactions.

Abstract

Socially Interactive Agents (SIAs) are physical or virtual embodied agents that display similar behavior as human multimodal behavior. Modeling SIAs' non-verbal behavior, such as speech and facial gestures, has always been a challenging task, given that a SIA can take the role of a speaker or a listener. A SIA must emit appropriate behavior adapted to its own speech, its previous behaviors (intra-personal), and the User's behaviors (inter-personal) for both roles. We propose AMII, a novel approach to synthesize adaptive facial gestures for SIAs while interacting with Users and acting interchangeably as a speaker or as a listener. AMII is characterized by modality memory encoding schema - where modality corresponds to either speech or facial gestures - and makes use of attention mechanisms to capture the intra-personal and inter-personal relationships. We validate our approach by conducting objective evaluations and comparing it with the state-of-the-art approaches.