To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures

TL;DR

This study compares rating and pairwise comparison methods for evaluating co-speech gestures in avatars, finding both effective but pairwise comparisons are faster and more reliable for small-scale assessments.

Contribution

It provides an empirical comparison of rating versus pairwise comparison methods for subjective evaluation of artificial human-like behaviour.

Findings

Both methods effectively rank gesture quality.

Pairwise comparisons are slightly faster.

Pairwise comparisons have better inter-rater reliability.

Abstract

While automatic performance metrics are crucial for machine learning of artificial human-like behaviour, the gold standard for evaluation remains human judgement. The subjective evaluation of artificial human-like behaviour in embodied conversational agents is however expensive and little is known about the quality of the data it returns. Two approaches to subjective evaluation can be largely distinguished, one relying on ratings, the other on pairwise comparisons. In this study we use co-speech gestures to compare the two against each other and answer questions about their appropriateness for evaluation of artificial behaviour. We consider their ability to rate quality, but also aspects pertaining to the effort of use and the time required to collect subjective data. We use crowd sourcing to rate the quality of co-speech gestures in avatars, assessing which method picks up more detail in subjective assessments. We compared gestures generated by three different machine learning models with various level of behavioural quality. We found that both approaches were able to rank the videos according to quality and that the ranking significantly correlated, showing that in terms of quality there is no preference of one method over the other. We also found that pairwise comparisons were slightly faster and came with improved inter-rater reliability, suggesting that for small-scale studies pairwise comparisons are to be favoured over ratings.