InsTaG: Learning Personalized 3D Talking Head from Few-Second Video

TL;DR

InsTaG is a novel framework for quickly synthesizing realistic personalized 3D talking heads from limited data, combining a lightweight person-specific model with universal motion priors for efficient adaptation.

Contribution

The paper introduces InsTaG, a fast-learning 3D talking head system using identity-free pre-training and motion-aligned adaptation for personalized synthesis with minimal data.

Findings

High-quality talking head synthesis with few training samples

Fast adaptation to new identities

Outperforms existing methods in efficiency and realism

Abstract

Despite exhibiting impressive performance in synthesizing lifelike personalized 3D talking heads, prevailing methods based on radiance fields suffer from high demands for training data and time for each new identity. This paper introduces InsTaG, a 3D talking head synthesis framework that allows a fast learning of realistic personalized 3D talking head from few training data. Built upon a lightweight 3DGS person-specific synthesizer with universal motion priors, InsTaG achieves high-quality and fast adaptation while preserving high-level personalization and efficiency. As preparation, we first propose an Identity-Free Pre-training strategy that enables the pre-training of the person-specific model and encourages the collection of universal motion priors from long-video data corpus. To fully exploit the universal motion priors to learn an unseen new identity, we then present a Motion-Aligned Adaptation strategy to adaptively align the target head to the pre-trained field, and constrain a robust dynamic head structure under few training data. Experiments demonstrate our outstanding performance and efficiency under various data scenarios to render high-quality personalized talking heads.