Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatars

TL;DR

Audio2Face-3D is a real-time, open-source system that generates realistic facial animations for digital avatars driven by audio input, enhancing interactive experiences and avatar creation.

Contribution

This paper introduces NVIDIA Audio2Face-3D, a comprehensive system with open-source tools for audio-driven facial animation of digital avatars, including data, architecture, and evaluation methods.

Findings

Enables real-time facial animation for avatars

Provides open-source SDK and training framework

Facilitates realistic avatar interaction

Abstract

Audio-driven facial animation presents an effective solution for animating digital avatars. In this paper, we detail the technical aspects of NVIDIA Audio2Face-3D, including data acquisition, network architecture, retargeting methodology, evaluation metrics, and use cases. Audio2Face-3D system enables real-time interaction between human users and interactive avatars, facilitating facial animation authoring for game characters. To assist digital avatar creators and game developers in generating realistic facial animations, we have open-sourced Audio2Face-3D networks, SDK, training framework, and example dataset.