Metric Design != Metric Behavior: Improving Metric Selection for the Unbiased Evaluation of Dimensionality Reduction

TL;DR

This paper introduces a new workflow for selecting diverse evaluation metrics for dimensionality reduction, reducing bias caused by correlated metrics and improving the stability of evaluation results.

Contribution

The paper proposes a clustering-based metric selection workflow that minimizes correlation bias in DR evaluation, enhancing assessment reliability.

Findings

Clustering metrics based on correlation improves evaluation stability.

The workflow reduces bias from correlated metrics.

Experimental results show increased robustness in DR evaluation.

Abstract

Evaluating the accuracy of dimensionality reduction (DR) projections in preserving the structure of high-dimensional data is crucial for reliable visual analytics. Diverse evaluation metrics targeting different structural characteristics have thus been developed. However, evaluations of DR projections can become biased if highly correlated metrics--those measuring similar structural characteristics--are inadvertently selected, favoring DR techniques that emphasize those characteristics. To address this issue, we propose a novel workflow that reduces bias in the selection of evaluation metrics by clustering metrics based on their empirical correlations rather than on their intended design characteristics alone. Our workflow works by computing metric similarity using pairwise correlations, clustering metrics to minimize overlap, and selecting a representative metric from each cluster. Quantitative experiments demonstrate that our approach improves the stability of DR evaluation, which indicates that our workflow contributes to mitigating evaluation bias.