Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis

TL;DR

This paper proposes a standardized, evolving benchmark and leaderboard for evaluating conversational motion synthesis, aiming to improve comparability and drive progress in gesture generation research.

Contribution

It introduces a living, community-driven leaderboard for gesture synthesis evaluation, addressing current inconsistencies and fostering ongoing progress.

Findings

Identifies issues with current gesture evaluation methods.

Proposes a scalable, community-updated benchmarking platform.

Aims to standardize and improve gesture synthesis assessment.

Abstract

Current evaluation practices in speech-driven gesture generation lack standardisation and focus on aspects that are easy to measure over aspects that actually matter. This leads to a situation where it is impossible to know what is the state of the art, or to know which method works better for which purpose when comparing two publications. In this position paper, we review and give details on issues with existing gesture-generation evaluation, and present a novel proposal for remedying them. Specifically, we announce an upcoming living leaderboard to benchmark progress in conversational motion synthesis. Unlike earlier gesture-generation challenges, the leaderboard will be updated with large-scale user studies of new gesture-generation systems multiple times per year, and systems on the leaderboard can be submitted to any publication venue that their authors prefer. By evolving the leaderboard evaluation data and tasks over time, the effort can keep driving progress towards the most important end goals identified by the community. We actively seek community involvement across the entire evaluation pipeline: from data and tasks for the evaluation, via tooling, to the systems evaluated. In other words, our proposal will not only make it easier for researchers to perform good evaluations, but their collective input and contributions will also help drive the future of gesture-generation research.