Robust One Shot Audio to Video Generation

TL;DR

This paper introduces OneShotA2V, a novel method for generating high-quality talking-head videos from a single image and audio, using curriculum and few-shot learning to adapt to unseen persons with multilingual capabilities.

Contribution

The paper presents a new approach combining curriculum learning and few-shot adaptation for one-shot audio-to-video generation of talking heads, outperforming existing methods.

Findings

Superior quantitative metrics (SSIM, PSNR, CPBD) compared to prior methods.

Qualitative and Turing tests confirm high realism and effectiveness.

Multilingual applicability demonstrated across diverse audio inputs.

Abstract

Audio to Video generation is an interesting problem that has numerous applications across industry verticals including film making, multi-media, marketing, education and others. High-quality video generation with expressive facial movements is a challenging problem that involves complex learning steps for generative adversarial networks. Further, enabling one-shot learning for an unseen single image increases the complexity of the problem while simultaneously making it more applicable to practical scenarios. In the paper, we propose a novel approach OneShotA2V to synthesize a talking person video of arbitrary length using as input: an audio signal and a single unseen image of a person. OneShotA2V leverages curriculum learning to learn movements of expressive facial components and hence generates a high-quality talking-head video of the given person. Further, it feeds the features generated from the audio input directly into a generative adversarial network and it adapts to any given unseen selfie by applying fewshot learning with only a few output updation epochs. OneShotA2V leverages spatially adaptive normalization based multi-level generator and multiple multi-level discriminators based architecture. The input audio clip is not restricted to any specific language, which gives the method multilingual applicability. Experimental evaluation demonstrates superior performance of OneShotA2V as compared to Realistic Speech-Driven Facial Animation with GANs(RSDGAN) [43], Speech2Vid [8], and other approaches, on multiple quantitative metrics including: SSIM (structural similarity index), PSNR (peak signal to noise ratio) and CPBD (image sharpness). Further, qualitative evaluation and Online Turing tests demonstrate the efficacy of our approach.