Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark

TL;DR

This paper highlights the lack of standardised human evaluation practices in speech-driven gesture generation, introduces a detailed protocol for the BEAT2 dataset, and provides comprehensive benchmark results and resources to improve future research.

Contribution

It proposes a standardized human evaluation protocol for gesture generation, conducts a large-scale benchmark across models, and releases datasets and tools to facilitate consistent future assessments.

Findings

Motion realism is saturated; older models perform as well as newer ones.

High speech-gesture alignment claims do not hold under rigorous evaluation.

Disentangled assessments of motion quality and alignment are necessary for accurate benchmarking.

Abstract

We review human evaluation practices in automatic, speech-driven 3D gesture generation and find a lack of standardisation and frequent use of flawed experimental setups. This leads to a situation where it is impossible to know how different methods compare, or what the state of the art is. In order to address common shortcomings of evaluation design, and to standardise future user studies in gesture-generation works, we introduce a detailed human evaluation protocol for the widely-used BEAT2 motion-capture dataset. Using this protocol, we conduct large-scale crowdsourced evaluation to rank six recent gesture-generation models -- each trained by its original authors -- across two key evaluation dimensions: motion realism and speech-gesture alignment. Our results show that 1) motion realism has become a saturated evaluation measure on the BEAT2 dataset, with older models performing on par with more recent approaches; 2) previous findings of high speech-gesture alignment do not hold up under rigorous evaluation, even for specialised models; and 3) the field must adopt disentangled assessments of motion quality and multimodal alignment for accurate benchmarking in order to make progress. To drive standardisation and enable new evaluation research, we release five hours of synthetic motion from the benchmarked models; over 750 rendered video stimuli from the user studies -- enabling new evaluations without requiring model reimplementation -- alongside our open-source rendering script, and 16,000 pairwise human preference votes collected for our benchmark.