Language-Guided Face Animation by Recurrent StyleGAN-based Generator

TL;DR

This paper introduces a novel framework for language-guided face animation that leverages a recurrent motion generator and StyleGAN, enabling realistic video synthesis from a single image guided by natural language.

Contribution

It proposes a new method combining language semantics and motions with a recurrent generator and StyleGAN for high-quality face animation.

Findings

Effective in animating diverse face types including human, anime, and dog faces.

Produces high-quality, realistic videos from static images guided by language.

Outperforms existing methods in both qualitative and quantitative evaluations.

Abstract

Recent works on language-guided image manipulation have shown great power of language in providing rich semantics, especially for face images. However, the other natural information, motions, in language is less explored. In this paper, we leverage the motion information and study a novel task, language-guided face animation, that aims to animate a static face image with the help of languages. To better utilize both semantics and motions from languages, we propose a simple yet effective framework. Specifically, we propose a recurrent motion generator to extract a series of semantic and motion information from the language and feed it along with visual information to a pre-trained StyleGAN to generate high-quality frames. To optimize the proposed framework, three carefully designed loss functions are proposed including a regularization loss to keep the face identity, a path length regularization loss to ensure motion smoothness, and a contrastive loss to enable video synthesis with various language guidance in one single model. Extensive experiments with both qualitative and quantitative evaluations on diverse domains (\textit{e.g.,} human face, anime face, and dog face) demonstrate the superiority of our model in generating high-quality and realistic videos from one still image with the guidance of language. Code will be available at https://github.com/TiankaiHang/language-guided-animation.git.