EvalTalker: Learning to Evaluate Real-Portrait-Driven Multi-Subject Talking Humans

TL;DR

This paper introduces EvalTalker, a new framework for assessing the quality of multi-subject talking human videos, addressing current limitations in multi-talker generation by leveraging a large-scale dataset and perceptual analysis.

Contribution

It presents the first large-scale Multi-Talker-generated Talking Human Quality Assessment dataset and a novel evaluation framework that correlates well with human subjective scores.

Findings

EvalTalker outperforms existing methods in correlation with subjective scores.

Identified 12 common distortion types affecting multi-talker quality.

Constructed a dataset of 5,492 multi-talker generated talking humans.

Abstract

Speech-driven Talking Human (TH) generation, commonly known as "Talker," currently faces limitations in multi-subject driving capabilities. Extending this paradigm to "Multi-Talker," capable of animating multiple subjects simultaneously, introduces richer interactivity and stronger immersion in audiovisual communication. However, current Multi-Talkers still exhibit noticeable quality degradation caused by technical limitations, resulting in suboptimal user experiences. To address this challenge, we construct THQA-MT, the first large-scale Multi-Talker-generated Talking Human Quality Assessment dataset, consisting of 5,492 Multi-Talker-generated THs (MTHs) from 15 representative Multi-Talkers using 400 real portraits collected online. Through subjective experiments, we analyze perceptual discrepancies among different Multi-Talkers and identify 12 common types of distortion. Furthermore, we introduce EvalTalker, a novel TH quality assessment framework. This framework possesses the ability to perceive global quality, human characteristics, and identity consistency, while integrating Qwen-Sync to perceive multimodal synchrony. Experimental results demonstrate that EvalTalker achieves superior correlation with subjective scores, providing a robust foundation for future research on high-quality Multi-Talker generation and evaluation.