Democratizing the Creation of Animatable Facial Avatars

TL;DR

This paper introduces a democratized pipeline for creating personalized animatable facial avatars without high-end hardware, using image warping and projection techniques to reconstruct geometry and texture from real-world images.

Contribution

A novel method that leverages image warping and projection to build personalized facial avatars without expensive equipment, enabling easier and more accessible avatar creation.

Findings

Effective reconstruction of geometry for multiple expressions

Ability to create personalized animation rigs

Improved avatar quality post-import

Abstract

In high-end visual effects pipelines, a customized (and expensive) light stage system is (typically) used to scan an actor in order to acquire both geometry and texture for various expressions. Aiming towards democratization, we propose a novel pipeline for obtaining geometry and texture as well as enough expression information to build a customized person-specific animation rig without using a light stage or any other high-end hardware (or manual cleanup). A key novel idea consists of warping real-world images to align with the geometry of a template avatar and subsequently projecting the warped image into the template avatar's texture; importantly, this allows us to leverage baked-in real-world lighting/texture information in order to create surrogate facial features (and bridge the domain gap) for the sake of geometry reconstruction. Not only can our method be used to obtain a neutral expression geometry and de-lit texture, but it can also be used to improve avatars after they have been imported into an animation system (noting that such imports tend to be lossy, while also hallucinating various features). Since a default animation rig will contain template expressions that do not correctly correspond to those of a particular individual, we use a Simon Says approach to capture various expressions and build a person-specific animation rig (that moves like they do). Our aforementioned warping/projection method has high enough efficacy to reconstruct geometry corresponding to each expressions.