S^3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis

TL;DR

This paper introduces S^3D-NeRF, a novel single-shot neural radiance field approach that directly uses speech audio for high-fidelity talking head synthesis, improving lip synchronization and visual quality.

Contribution

The paper proposes a hierarchical appearance encoder, a cross-modal deformation field, and a lip-sync discriminator to enable direct speech-driven face synthesis with enhanced realism and synchronization.

Findings

Outperforms previous methods in video fidelity.

Achieves superior audio-lip synchronization.

Maintains temporal consistency in lip movements.

Abstract

Talking head synthesis is a practical technique with wide applications. Current Neural Radiance Field (NeRF) based approaches have shown their superiority on driving one-shot talking heads with videos or signals regressed from audio. However, most of them failed to take the audio as driven information directly, unable to enjoy the flexibility and availability of speech. Since mapping audio signals to face deformation is non-trivial, we design a Single-Shot Speech-Driven Neural Radiance Field (S^3D-NeRF) method in this paper to tackle the following three difficulties: learning a representative appearance feature for each identity, modeling motion of different face regions with audio, and keeping the temporal consistency of the lip area. To this end, we introduce a Hierarchical Facial Appearance Encoder to learn multi-scale representations for catching the appearance of different speakers, and elaborate a Cross-modal Facial Deformation Field to perform speech animation according to the relationship between the audio signal and different face regions. Moreover, to enhance the temporal consistency of the important lip area, we introduce a lip-sync discriminator to penalize the out-of-sync audio-visual sequences. Extensive experiments have shown that our S^3D-NeRF surpasses previous arts on both video fidelity and audio-lip synchronization.