Evaluating the Evaluators: Towards Human-aligned Metrics for Missing Markers Reconstruction

TL;DR

This paper critiques the common use of mean square error for evaluating missing marker reconstruction in animation data and proposes new metrics that better align with human perception, advancing the field.

Contribution

It introduces and evaluates new evaluation metrics that correlate more closely with subjective perception, improving the assessment of missing marker reconstruction methods.

Findings

Mean square error does not align with human perception.

New metrics show better correlation with subjective quality.

Proposed metrics can guide future improvements.

Abstract

Animation data is often obtained through optical motion capture systems, which utilize a multitude of cameras to establish the position of optical markers. However, system errors or occlusions can result in missing markers, the manual cleaning of which can be time-consuming. This has sparked interest in machine learning-based solutions for missing marker reconstruction in the academic community. Most academic papers utilize a simplistic mean square error as the main metric. In this paper, we show that this metric does not correlate with subjective perception of the fill quality. Additionally, we introduce and evaluate a set of better-correlated metrics that can drive progress in the field.