Predicting Out-of-Distribution Error with Confidence Optimal Transport

TL;DR

This paper introduces Confidence Optimal Transport (COT), a novel method based on optimal transport theory, to accurately predict a model's performance on out-of-distribution data without additional annotations, achieving state-of-the-art results.

Contribution

The paper proposes a simple, effective approach using optimal transport to estimate model performance on unknown distributions without extra labels, advancing OOD performance prediction.

Findings

COT provides robust performance estimates on OOD data.

COT outperforms existing methods by a large margin.

Achieves state-of-the-art results on three benchmark datasets.

Abstract

Out-of-distribution (OOD) data poses serious challenges in deployed machine learning models as even subtle changes could incur significant performance drops. Being able to estimate a model's performance on test data is important in practice as it indicates when to trust to model's decisions. We present a simple yet effective method to predict a model's performance on an unknown distribution without any addition annotation. Our approach is rooted in the Optimal Transport theory, viewing test samples' output softmax scores from deep neural networks as empirical samples from an unknown distribution. We show that our method, Confidence Optimal Transport (COT), provides robust estimates of a model's performance on a target domain. Despite its simplicity, our method achieves state-of-the-art results on three benchmark datasets and outperforms existing methods by a large margin.