FreeTalk: Emotional Topology-Free 3D Talking Heads

TL;DR

FreeTalk is a novel framework for emotion-conditioned 3D talking-head animation that generalizes to arbitrary mesh topologies and captures both articulatory and emotional facial motions without relying on template meshes.

Contribution

It introduces a two-stage process combining sparse landmark prediction from speech with topology-free mesh deformation, enabling robust animation on raw 3D scans with arbitrary topology.

Findings

Matches specialized baselines in-domain

Improves robustness to unseen identities

Operates without template fitting or correspondence supervision

Abstract

Speech-driven 3D facial animation has advanced rapidly, yet most approaches remain tied to registered template meshes, preventing effective deployment on raw 3D scans with arbitrary topology. At the same time, modeling controllable emotional dynamics beyond lip articulation remains challenging, and is often tied to template-based parameterizations. We address these challenges by proposing FreeTalk, a two-stage framework for emotion-conditioned 3D talking-head animation that generalizes to unregistered face meshes with arbitrary vertex count and connectivity. First, Audio-To-Sparse (ATS) predicts a temporally coherent sequence of 3D landmark displacements from speech audio, conditioned on an emotion category and intensity. This sparse representation captures both articulatory and affective motion while remaining independent of mesh topology. Second, Sparse-To-Mesh (STM) transfers the predicted landmark motion to a target mesh by combining intrinsic surface features with landmark-to-vertex conditioning, producing dense per-vertex deformations without template fitting or correspondence supervision at test time. Extensive experiments show that FreeTalk matches specialized baselines when trained in-domain, while providing substantially improved robustness to unseen identities and mesh topologies. Code and pre-trained models will be made publicly available.