Alignment Scores: Robust Metrics for Multiview Pose Accuracy Evaluation

TL;DR

This paper introduces three new metrics—TAS, RAS, and PAS—for evaluating multiview pose accuracy, emphasizing robustness to outliers and independence of translation and rotation assessments.

Contribution

The paper presents novel, robust metrics for pose accuracy evaluation that outperform existing methods in handling outliers and parameter adjustments.

Findings

TAS and RAS are more robust to outliers than existing metrics.

The proposed metrics do not require dataset-specific parameter tuning.

Extensive simulations validate the effectiveness of the new metrics.

Abstract

We propose three novel metrics for evaluating the accuracy of a set of estimated camera poses given the ground truth: Translation Alignment Score (TAS), Rotation Alignment Score (RAS), and Pose Alignment Score (PAS). The TAS evaluates the translation accuracy independently of the rotations, and the RAS evaluates the rotation accuracy independently of the translations. The PAS is the average of the two scores, evaluating the combined accuracy of both translations and rotations. The TAS is computed in four steps: (1) Find the upper quartile of the closest-pair-distances, $d$. (2) Align the estimated trajectory to the ground truth using a robust registration method. (3) Collect all distance errors and obtain the cumulative frequencies for multiple thresholds ranging from $0.01d$ to $d$ with a resolution $0.01d$. (4) Add up these cumulative frequencies and normalize them such that the theoretical maximum is 1. The TAS has practical advantages over the existing metrics in that (1) it is robust to outliers and collinear motion, and (2) there is no need to adjust parameters on different datasets. The RAS is computed in a similar manner to the TAS and is also shown to be more robust against outliers than the existing rotation metrics. We verify our claims through extensive simulations and provide in-depth discussion of the strengths and weaknesses of the proposed metrics.