Who is a Better Talker: Subjective and Objective Quality Assessment for AI-Generated Talking Heads

TL;DR

This paper introduces a large dataset for assessing AI-generated talking heads, analyzes their quality subjectively, and proposes an objective assessment method that achieves state-of-the-art results.

Contribution

It presents the largest AGTH quality dataset, evaluates talker performance, and develops a novel objective quality assessment method.

Findings

The dataset contains 10,457 AGTHs for comprehensive analysis.

The subjective evaluation reveals common distortions in AGTHs.

The proposed method outperforms existing quality assessment techniques.

Abstract

Speech-driven methods for portraits are figuratively known as "Talkers" because of their capability to synthesize speaking mouth shapes and facial movements. Especially with the rapid development of the Text-to-Image (T2I) models, AI-Generated Talking Heads (AGTHs) have gradually become an emerging digital human media. However, challenges persist regarding the quality of these talkers and AGTHs they generate, and comprehensive studies addressing these issues remain limited. To address this gap, this paper presents the largest AGTH quality assessment dataset THQA-10K to date, which selects 12 prominent T2I models and 14 advanced talkers to generate AGTHs for 14 prompts. After excluding instances where AGTH generation is unsuccessful, the THQA-10K dataset contains 10,457 AGTHs. Then, volunteers are recruited to subjectively rate the AGTHs and give the corresponding distortion categories. In our analysis for subjective experimental results, we evaluate the performance of talkers in terms of generalizability and quality, and also expose the distortions of existing AGTHs. Finally, an objective quality assessment method based on the first frame, Y-T slice and tone-lip consistency is proposed. Experimental results show that this method can achieve state-of-the-art (SOTA) performance in AGTH quality assessment. The work is released at https://github.com/zyj-2000/Talker.