Learning and Evaluating Human Preferences for Conversational Head Generation

TL;DR

This paper introduces Preference Score, a learning-based metric that accurately predicts human preferences in conversational head video synthesis, enabling efficient and reliable evaluation without manual annotation.

Contribution

The paper presents a novel Preference Score metric that aligns with human preferences and can be used for automatic evaluation of conversational head generation.

Findings

Preference Score correlates well with human judgments.

The metric is robust across different datasets.

It reduces the need for labor-intensive user studies.

Abstract

A reliable and comprehensive evaluation metric that aligns with manual preference assessments is crucial for conversational head video synthesis methods development. Existing quantitative evaluations often fail to capture the full complexity of human preference, as they only consider limited evaluation dimensions. Qualitative evaluations and user studies offer a solution but are time-consuming and labor-intensive. This limitation hinders the advancement of conversational head generation algorithms and systems. In this paper, we propose a novel learning-based evaluation metric named Preference Score (PS) for fitting human preference according to the quantitative evaluations across different dimensions. PS can serve as a quantitative evaluation without the need for human annotation. Experimental results validate the superiority of Preference Score in aligning with human perception, and also demonstrate robustness and generalizability to unseen data, making it a valuable tool for advancing conversation head generation. We expect this metric could facilitate new advances in conversational head generation. Project Page: https://https://github.com/dc3ea9f/PreferenceScore.