Metric Learning Improves the Ability of Combinatorial Coverage Metrics to Anticipate Classification Error

TL;DR

This paper demonstrates that applying metric learning enhances combinatorial coverage metrics' ability to predict classification errors and detect out-of-distribution data across multiple datasets.

Contribution

The study introduces metric learning as a method to improve the predictive power of coverage metrics for classification errors, addressing dataset dependence issues.

Findings

Metric learning increases the difference between correct and incorrect classifications in coverage metrics.

Coverage metrics become more reliable in predicting errors after applying metric learning.

Statistical tests confirm the significance of improved predictive performance.

Abstract

Machine learning models are increasingly used in practice. However, many machine learning methods are sensitive to test or operational data that is dissimilar to training data. Out-of-distribution (OOD) data is known to increase the probability of error and research into metrics that identify what dissimilarities in data affect model performance is on-going. Recently, combinatorial coverage metrics have been explored in the literature as an alternative to distribution-based metrics. Results show that coverage metrics can correlate with classification error. However, other results show that the utility of coverage metrics is highly dataset-dependent. In this paper, we show that this dataset-dependence can be alleviated with metric learning, a machine learning technique for learning latent spaces where data from different classes is further apart. In a study of 6 open-source datasets, we find that metric learning increased the difference between set-difference coverage metrics (SDCCMs) calculated on correctly and incorrectly classified data, thereby demonstrating that metric learning improves the ability of SDCCMs to anticipate classification error. Paired t-tests validate the statistical significance of our findings. Overall, we conclude that metric learning improves the ability of coverage metrics to anticipate classifier error and identify when OOD data is likely to degrade model performance.