OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance

TL;DR

OpFlowTalker leverages optical flow guidance to generate realistic, temporally coherent talking face videos that improve lip-readability and semantic consistency, addressing previous limitations in frame-to-frame smoothness and visual quality.

Contribution

This paper introduces OpFlowTalker, a novel method using optical flow prediction from audio to enhance temporal coherence and semantic accuracy in talking face video synthesis.

Findings

Improved visual smoothness and lip-readability in synthesized videos.

Enhanced temporal coherence through sequence fusion and optical flow guidance.

Validated effectiveness with extensive empirical experiments.

Abstract

Creating realistic, natural, and lip-readable talking face videos remains a formidable challenge. Previous research primarily concentrated on generating and aligning single-frame images while overlooking the smoothness of frame-to-frame transitions and temporal dependencies. This often compromised visual quality and effects in practical settings, particularly when handling complex facial data and audio content, which frequently led to semantically incongruent visual illusions. Specifically, synthesized videos commonly featured disorganized lip movements, making them difficult to understand and recognize. To overcome these limitations, this paper introduces the application of optical flow to guide facial image generation, enhancing inter-frame continuity and semantic consistency. We propose "OpFlowTalker", a novel approach that utilizes predicted optical flow changes from audio inputs rather than direct image predictions. This method smooths image transitions and aligns changes with semantic content. Moreover, it employs a sequence fusion technique to replace the independent generation of single frames, thus preserving contextual information and maintaining temporal coherence. We also developed an optical flow synchronization module that regulates both full-face and lip movements, optimizing visual synthesis by balancing regional dynamics. Furthermore, we introduce a Visual Text Consistency Score (VTCS) that accurately measures lip-readability in synthesized videos. Extensive empirical evidence validates the effectiveness of our approach.