EmbedTalk: Triplane-Free Talking Head Synthesis using Embedding-Driven Gaussian Deformation

TL;DR

EmbedTalk introduces embedding-driven Gaussian deformation for real-time talking head synthesis, surpassing traditional tri-plane methods in quality, efficiency, and compactness, enabling high FPS on mobile GPUs.

Contribution

The paper presents EmbedTalk, a novel embedding-based approach for speech-driven facial deformation that improves over tri-plane representations in quality and computational efficiency.

Findings

Outperforms existing 3DGS methods in rendering quality and lip sync.

Achieves over 60 FPS on a mobile GPU with a compact model.

Replaces tri-plane encoding with learned embeddings for better efficiency.

Abstract

Real-time talking head synthesis increasingly relies on deformable 3D Gaussian Splatting (3DGS) due to its low latency. Tri-planes are the standard choice for encoding Gaussians prior to deformation, since they provide a continuous domain with explicit spatial relationships. However, tri-plane representations are limited by grid resolution and approximation errors introduced by projecting 3D volumetric fields onto 2D subspaces. Recent work has shown the superiority of learnt embeddings for driving temporal deformations in 4D scene reconstruction. We introduce $\textbf{EmbedTalk}$, which shows how such embeddings can be leveraged for modelling speech deformations in talking head synthesis. Through comprehensive experiments, we show that EmbedTalk outperforms existing 3DGS-based methods in rendering quality, lip synchronisation, and motion consistency, while remaining competitive with state-of-the-art generative models. Moreover, replacing the tri-plane encoding with learnt embeddings enables significantly more compact models that achieve over 60 FPS on a mobile GPU (RTX 2060 6 GB). Our code will be placed in the public domain on acceptance.