Learn2Talk: 3D Talking Face Learns from 2D Talking Face

TL;DR

Learn2Talk introduces a novel framework that leverages 2D talking face techniques to enhance 3D talking face models, significantly improving lip-sync accuracy, vertex precision, and speech perception in 3D facial animation.

Contribution

The paper presents a new learning framework that integrates 2D talking face expertise into 3D models, addressing the gap in lip-sync and speech perception quality.

Findings

Enhanced lip-sync accuracy compared to state-of-the-art methods.

Improved 3D vertex precision in facial animations.

Better speech perception in 3D talking face models.

Abstract

Speech-driven facial animation methods usually contain two main classes, 3D and 2D talking face, both of which attract considerable research attention in recent years. However, to the best of our knowledge, the research on 3D talking face does not go deeper as 2D talking face, in the aspect of lip-synchronization (lip-sync) and speech perception. To mind the gap between the two sub-fields, we propose a learning framework named Learn2Talk, which can construct a better 3D talking face network by exploiting two expertise points from the field of 2D talking face. Firstly, inspired by the audio-video sync network, a 3D sync-lip expert model is devised for the pursuit of lip-sync between audio and 3D facial motion. Secondly, a teacher model selected from 2D talking face methods is used to guide the training of the audio-to-3D motions regression network to yield more 3D vertex accuracy. Extensive experiments show the advantages of the proposed framework in terms of lip-sync, vertex accuracy and speech perception, compared with state-of-the-arts. Finally, we show two applications of the proposed framework: audio-visual speech recognition and speech-driven 3D Gaussian Splatting based avatar animation.