TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation

TL;DR

TempoSyncDiff introduces a distilled, temporally-consistent diffusion framework for low-latency, stable, and accurate audio-driven talking head generation suitable for edge devices, addressing key challenges in real-time human synthesis.

Contribution

The paper presents a novel reference-conditioned latent diffusion model with teacher-student distillation, identity anchoring, and temporal regularization for efficient, stable, and realistic talking-head synthesis.

Findings

Achieves lower latency inference with retained quality compared to standard diffusion models.

Reduces flicker and identity drift through temporal regularization and identity anchoring.

Demonstrates feasibility of edge deployment with CPU-only and edge computing measurements.

Abstract

Diffusion models have recently advanced photorealistic human synthesis, although practical talking-head generation (THG) remains constrained by high inference latency, temporal instability such as flicker and identity drift, and imperfect audio-visual alignment under challenging speech conditions. This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework that explores few-step inference for efficient audio-driven talking-head generation. The approach adopts a teacher-student distillation formulation in which a diffusion teacher trained with a standard noise prediction objective guides a lightweight student denoiser capable of operating with significantly fewer inference steps to improve generation stability. The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker during synthesis, while viseme-based audio conditioning provides coarse lip motion control. Experiments on the LRS3 dataset report denoising-stage component-level metrics relative to VAE reconstructions and preliminary latency characterization, including CPU-only and edge computing measurements and feasibility estimates for edge deployment. The results suggest that distilled diffusion models can retain much of the reconstruction behaviour of a stronger teacher while enabling substantially lower latency inference. The study is positioned as an initial step toward practical diffusion-based talking-head generation under constrained computational settings. GitHub: https://mazumdarsoumya.github.io/TempoSyncDiff