Source-Free Domain-Invariant Performance Prediction

TL;DR

This paper introduces a novel source-free method for predicting model performance across different domains by leveraging uncertainty estimation and generative calibration, outperforming existing approaches especially when source data is unavailable.

Contribution

The work presents a new source-free performance prediction method that uses uncertainty-based calibration and gradient evaluation, advancing the accuracy of domain-invariant performance estimation without source data.

Findings

Outperforms existing source-free methods in benchmarks

Significantly better than source-based methods with limited source data

Effective in domain-invariant performance prediction

Abstract

Accurately estimating model performance poses a significant challenge, particularly in scenarios where the source and target domains follow different data distributions. Most existing performance prediction methods heavily rely on the source data in their estimation process, limiting their applicability in a more realistic setting where only the trained model is accessible. The few methods that do not require source data exhibit considerably inferior performance. In this work, we propose a source-free approach centred on uncertainty-based estimation, using a generative model for calibration in the absence of source data. We establish connections between our approach for unsupervised calibration and temperature scaling. We then employ a gradient-based strategy to evaluate the correctness of the calibrated predictions. Our experiments on benchmark object recognition datasets reveal that existing source-based methods fall short with limited source sample availability. Furthermore, our approach significantly outperforms the current state-of-the-art source-free and source-based methods, affirming its effectiveness in domain-invariant performance estimation.