What is the Best Automated Metric for Text to Motion Generation?

TL;DR

This paper evaluates existing automated metrics for text-to-motion generation, finds their limitations, and introduces MoBERT, a new metric that aligns closely with human judgments at both sample and model levels.

Contribution

The paper systematically assesses current metrics' correlation with human judgments and proposes MoBERT, a novel multimodal BERT-based metric with superior alignment.

Findings

Current metrics poorly correlate with human judgments at sample level.

Common metrics like R-Precision correlate well at the model level.

MoBERT outperforms existing metrics in aligning with human evaluations.

Abstract

There is growing interest in generating skeleton-based human motions from natural language descriptions. While most efforts have focused on developing better neural architectures for this task, there has been no significant work on determining the proper evaluation metric. Human evaluation is the ultimate accuracy measure for this task, and automated metrics should correlate well with human quality judgments. Since descriptions are compatible with many motions, determining the right metric is critical for evaluating and designing effective generative models. This paper systematically studies which metrics best align with human evaluations and proposes new metrics that align even better. Our findings indicate that none of the metrics currently used for this task show even a moderate correlation with human judgments on a sample level. However, for assessing average model performance, commonly used metrics such as R-Precision and less-used coordinate errors show strong correlations. Additionally, several recently developed metrics are not recommended due to their low correlation compared to alternatives. We also introduce a novel metric based on a multimodal BERT-like model, MoBERT, which offers strongly human-correlated sample-level evaluations while maintaining near-perfect model-level correlation. Our results demonstrate that this new metric exhibits extensive benefits over all current alternatives.