SingingHead: A Large-scale 4D Dataset for Singing Head Animation

TL;DR

This paper introduces SingingHead, a large-scale dataset for singing head animation, and proposes UniSinger, a unified framework for 3D and 2D singing face synthesis, advancing audio-driven facial animation research.

Contribution

The paper provides the first large-scale singing head dataset and a unified model for 3D and 2D singing face animation, bridging the gap between singing and talking face synthesis.

Findings

SingingHead dataset contains 27 hours of synchronized singing videos from 76 individuals.

Benchmarking shows existing methods perform suboptimally on singing tasks.

UniSinger achieves competitive results in both 3D and 2D singing face synthesis.

Abstract

Singing, as a common facial movement second only to talking, can be regarded as a universal language across ethnicities and cultures, plays an important role in emotional communication, art, and entertainment. However, it is often overlooked in the field of audio-driven facial animation due to the lack of singing head datasets and the domain gap between singing and talking in rhythm and amplitude. To this end, we collect a high-quality large-scale singing head dataset, SingingHead, which consists of more than 27 hours of synchronized singing video, 3D facial motion, singing audio, and background music from 76 individuals and 8 types of music. Along with the SingingHead dataset, we benchmark existing audio-driven 3D facial animation methods and 2D talking head methods on the singing task. Furthermore, we argue that 3D and 2D facial animation tasks can be solved together, and propose a unified singing head animation framework named UniSinger to achieve both singing audio-driven 3D singing head animation and 2D singing portrait video synthesis, which achieves competitive results on both 3D and 2D benchmarks. Extensive experiments demonstrate the significance of the proposed singing-specific dataset in promoting the development of singing head animation tasks, as well as the promising performance of our unified facial animation framework.