TDMM-LM: Bridging Facial Understanding and Animation via Language Models

TL;DR

This paper introduces a novel approach that uses language models to understand and generate facial animations by creating a large synthetic dataset and framing facial motion as a language problem, enabling bidirectional tasks.

Contribution

The work is the first to treat facial-parameter modeling as a language task, bridging facial understanding and animation through large-scale synthetic data and language models.

Findings

Language models can interpret facial motion with strong generalization.

Language models can synthesize facial motion from text prompts effectively.

The approach establishes a unified framework for facial animation and understanding.

Abstract

Text-guided human body animation has advanced rapidly, yet facial animation lags due to the scarcity of well-annotated, text-paired facial corpora. To close this gap, we leverage foundation generative models to synthesize a large, balanced corpus of facial behavior. We design prompts suite covering emotions and head motions, generate about 80 hours of facial videos with multiple generators, and fit per-frame 3D facial parameters, yielding large-scale (prompt and parameter) pairs for training. Building on this dataset, we probe language models for bidirectional competence over facial motion via two complementary tasks: (1) Motion2Language: given a sequence of 3D facial parameters, the model produces natural-language descriptions capturing content, style, and dynamics; and (2) Language2Motion: given a prompt, the model synthesizes the corresponding sequence of 3D facial parameters via quantized motion tokens for downstream animation. Extensive experiments show that in this setting language models can both interpret and synthesize facial motion with strong generalization. To best of our knowledge, this is the first work to cast facial-parameter modeling as a language problem, establishing a unified path for text-conditioned facial animation and motion understanding.