EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions

TL;DR

EchoMimic is a novel portrait animation method that combines audio and facial landmarks during training, resulting in more stable and natural lifelike video generation compared to existing approaches.

Contribution

It introduces a training strategy that concurrently uses audio and facial landmarks, enabling flexible and improved portrait animation.

Findings

Outperforms existing methods in quantitative metrics

Produces more natural and stable portrait videos

Works effectively with combined audio and landmark inputs

Abstract

The area of portrait image animation, propelled by audio input, has witnessed notable progress in the generation of lifelike and dynamic portraits. Conventional methods are limited to utilizing either audios or facial key points to drive images into videos, while they can yield satisfactory results, certain issues exist. For instance, methods driven solely by audios can be unstable at times due to the relatively weaker audio signal, while methods driven exclusively by facial key points, although more stable in driving, can result in unnatural outcomes due to the excessive control of key point information. In addressing the previously mentioned challenges, in this paper, we introduce a novel approach which we named EchoMimic. EchoMimic is concurrently trained using both audios and facial landmarks. Through the implementation of a novel training strategy, EchoMimic is capable of generating portrait videos not only by audios and facial landmarks individually, but also by a combination of both audios and selected facial landmarks. EchoMimic has been comprehensively compared with alternative algorithms across various public datasets and our collected dataset, showcasing superior performance in both quantitative and qualitative evaluations. Additional visualization and access to the source code can be located on the EchoMimic project page.