The GENEA Challenge 2023: A large scale evaluation of gesture generation models in monadic and dyadic settings

TL;DR

The GENEA Challenge 2023 evaluated speech-driven gesture generation models in monadic and dyadic settings, revealing significant variability in human-likeness and highlighting the challenges in achieving natural and contextually appropriate gestures.

Contribution

This paper presents a large-scale evaluation of gesture generation models in dyadic interactions, providing insights into their performance and the gap to natural motion.

Findings

Wide variation in human-likeness among submissions

Most systems perform slightly above chance in appropriateness

Dyadic appropriateness is challenging and not strongly correlated with speech quality

Abstract

This paper reports on the GENEA Challenge 2023, in which participating teams built speech-driven gesture-generation systems using the same speech and motion dataset, followed by a joint evaluation. This year's challenge provided data on both sides of a dyadic interaction, allowing teams to generate full-body motion for an agent given its speech (text and audio) and the speech and motion of the interlocutor. We evaluated 12 submissions and 2 baselines together with held-out motion-capture data in several large-scale user studies. The studies focused on three aspects: 1) the human-likeness of the motion, 2) the appropriateness of the motion for the agent's own speech whilst controlling for the human-likeness of the motion, and 3) the appropriateness of the motion for the behaviour of the interlocutor in the interaction, using a setup that controls for both the human-likeness of the motion and the agent's own speech. We found a large span in human-likeness between challenge submissions, with a few systems rated close to human mocap. Appropriateness seems far from being solved, with most submissions performing in a narrow range slightly above chance, far behind natural motion. The effect of the interlocutor is even more subtle, with submitted systems at best performing barely above chance. Interestingly, a dyadic system being highly appropriate for agent speech does not necessarily imply high appropriateness for the interlocutor. Additional material is available via the project website at https://svito-zar.github.io/GENEAchallenge2023/ .