A High-Fidelity Open Embodied Avatar with Lip Syncing and Expression Capabilities

TL;DR

This paper introduces an open-source, high-fidelity embodied avatar built with Unreal Engine, capable of lip syncing, facial expressions, and head gestures, controllable via a simple Python interface for social interaction applications.

Contribution

The work presents a customizable, open-architecture avatar with integrated lip sync and expression capabilities, along with accessible code and models for easy deployment and experimentation.

Findings

Realistic lip syncing and facial expressions achieved

Open-source code facilitates easy control and customization

Supports integration into conversational agents and social applications

Abstract

Embodied avatars as virtual agents have many applications and provide benefits over disembodied agents, allowing non-verbal social and interactional cues to be leveraged, in a similar manner to how humans interact with each other. We present an open embodied avatar built upon the Unreal Engine that can be controlled via a simple python programming interface. The avatar has lip syncing (phoneme control), head gesture and facial expression (using either facial action units or cardinal emotion categories) capabilities. We release code and models to illustrate how the avatar can be controlled like a puppet or used to create a simple conversational agent using public application programming interfaces (APIs). GITHUB link: https://github.com/danmcduff/AvatarSim