KeyframeFace: Language-Driven Facial Animation via Semantic Keyframes

TL;DR

KeyframeFace introduces a novel language-driven facial animation framework using semantic keyframes, enhancing control, interpretability, and editing precision over traditional dense motion regression methods.

Contribution

The paper proposes a semantic keyframe-based approach for facial animation driven by language, leveraging large language models and a new multimodal dataset for improved fidelity and interpretability.

Findings

Semantic keyframe supervision improves expression fidelity.

Language priors enhance semantic alignment of facial animations.

Constructed a multimodal dataset with 2,100 scripts and annotated keyframes.

Abstract

Facial animation is a core component for creating digital characters in Computer Graphics (CG) industry. A typical production workflow relies on sparse, semantically meaningful keyframes to precisely control facial expressions. Enabling such animation directly from natural-language descriptions could significantly improve content creation efficiency and accessibility. However, most existing methods adopt a text-to-continuous-frames paradigm, directly regressing dense facial motion trajectories from language. This formulation entangles high-level semantic intent with low-level motion, lacks explicit semantic control structure, and limits precise editing and interpretability. Inspired by the keyframe paradigm in animation production, we propose KeyframeFace, a framework for semantic facial animation from language via interpretable keyframes. Instead of predicting dense motion trajectories, our method represents animation as a sequence of semantically meaningful keyframes in an interpretable ARKit-based facial control space. A language-driven model leverages large language model (LLM) priors to generate keyframes that align with contextual text descriptions and emotion cues. To support this formulation, we construct a multimodal dataset comprising 2,100 expression scripts paired with monocular videos, per-frame ARKit coefficients, and manually annotated semantic keyframes. Experiments show that incorporating semantic keyframe supervision and language priors significantly improves expression fidelity and semantic alignment compared to methods that do not use facial action semantics.