Embedded Representation Learning Network for Animating Styled Video Portrait

TL;DR

This paper introduces ERLNet, a novel two-stage generative framework combining audio-driven facial expression synthesis and a dual-branch fusion NeRF to produce realistic, style-controllable talking head videos with reduced artifacts.

Contribution

The paper proposes ERLNet, a new generative paradigm with two stages that improves style control and reduces artifacts in talking head synthesis using NeRF-based methods.

Findings

Produces more realistic talking head videos.

Effectively controls style and expression.

Reduces displacement artifacts around the neck.

Abstract

The talking head generation recently attracted considerable attention due to its widespread application prospects, especially for digital avatars and 3D animation design. Inspired by this practical demand, several works explored Neural Radiance Fields (NeRF) to synthesize the talking heads. However, these methods based on NeRF face two challenges: (1) Difficulty in generating style-controllable talking heads. (2) Displacement artifacts around the neck in rendered images. To overcome these two challenges, we propose a novel generative paradigm \textit{Embedded Representation Learning Network} (ERLNet) with two learning stages. First, the \textit{ audio-driven FLAME} (ADF) module is constructed to produce facial expression and head pose sequences synchronized with content audio and style video. Second, given the sequence deduced by the ADF, one novel \textit{dual-branch fusion NeRF} (DBF-NeRF) explores these contents to render the final images. Extensive empirical studies demonstrate that the collaboration of these two stages effectively facilitates our method to render a more realistic talking head than the existing algorithms.