Human-like Nonverbal Behavior with MetaHumans in Real-World Interaction Studies: An Architecture Using Generative Methods and Motion Capture

TL;DR

This paper introduces a novel architecture combining MetaHumans, AI, and motion capture to enable human-like nonverbal behavior in virtual agents for real-world social interactions, demonstrated through a field study.

Contribution

It presents a new distributed system architecture integrating generative methods and motion capture for realistic nonverbal behavior in virtual agents.

Findings

Successful deployment in a three-week field study

Enhanced realism of nonverbal behaviors in virtual agents

Potential for research in authentic social interactions

Abstract

Socially interactive agents are gaining prominence in domains like healthcare, education, and service contexts, particularly virtual agents due to their inherent scalability. To facilitate authentic interactions, these systems require verbal and nonverbal communication through e.g., facial expressions and gestures. While natural language processing technologies have rapidly advanced, incorporating human-like nonverbal behavior into real-world interaction contexts is crucial for enhancing the success of communication, yet this area remains underexplored. One barrier is creating autonomous systems with sophisticated conversational abilities that integrate human-like nonverbal behavior. This paper presents a distributed architecture using Epic Games MetaHuman, combined with advanced conversational AI and camera-based user management, that supports methods like motion capture, handcrafted animation, and generative approaches for nonverbal behavior. We share insights into a system architecture designed to investigate nonverbal behavior in socially interactive agents, deployed in a three-week field study in the Deutsches Museum Bonn, showcasing its potential in realistic nonverbal behavior research.