One Shot Audio to Animated Video Generation

TL;DR

This paper introduces OneShotAu2AV, a novel two-stage method for generating animated videos from audio and a single image, capable of producing synchronized lip movements, facial expressions, and head motions for arbitrary-length videos.

Contribution

The paper presents a new two-stage approach that converts audio and a single image into animated videos, including a novel unsupervised domain translation from human to animated domain.

Findings

Outperforms U-GAT-IT and RecycleGan on multiple metrics

Generates lip-synced, expressive animated videos of arbitrary length

Applicable to multiple languages without restrictions

Abstract

We consider the challenging problem of audio to animated video generation. We propose a novel method OneShotAu2AV to generate an animated video of arbitrary length using an audio clip and a single unseen image of a person as an input. The proposed method consists of two stages. In the first stage, OneShotAu2AV generates the talking-head video in the human domain given an audio and a person's image. In the second stage, the talking-head video from the human domain is converted to the animated domain. The model architecture of the first stage consists of spatially adaptive normalization based multi-level generator and multiple multilevel discriminators along with multiple adversarial and non-adversarial losses. The second stage leverages attention based normalization driven GAN architecture along with temporal predictor based recycle loss and blink loss coupled with lipsync loss, for unsupervised generation of animated video. In our approach, the input audio clip is not restricted to any specific language, which gives the method multilingual applicability. OneShotAu2AV can generate animated videos that have: (a) lip movements that are in sync with the audio, (b) natural facial expressions such as blinks and eyebrow movements, (c) head movements. Experimental evaluation demonstrates superior performance of OneShotAu2AV as compared to U-GAT-IT and RecycleGan on multiple quantitative metrics including KID(Kernel Inception Distance), Word error rate, blinks/sec