Unsupervised Accuracy Estimation of Deep Visual Models using Domain-Adaptive Adversarial Perturbation without Source Samples

TL;DR

This paper introduces a source-free method for estimating the accuracy of deep visual models on unlabeled target data by using adaptive adversarial perturbations and pseudo-labels, addressing distribution shift without source data access.

Contribution

It proposes a novel source-free accuracy estimation framework leveraging adversarial perturbations and pseudo-label disagreement, outperforming existing source-dependent methods.

Findings

Effective accuracy estimation without source data

Outperforms existing source-dependent methods

Robust to distribution shifts

Abstract

Deploying deep visual models can lead to performance drops due to the discrepancies between source and target distributions. Several approaches leverage labeled source data to estimate target domain accuracy, but accessing labeled source data is often prohibitively difficult due to data confidentiality or resource limitations on serving devices. Our work proposes a new framework to estimate model accuracy on unlabeled target data without access to source data. We investigate the feasibility of using pseudo-labels for accuracy estimation and evolve this idea into adopting recent advances in source-free domain adaptation algorithms. Our approach measures the disagreement rate between the source hypothesis and the target pseudo-labeling function, adapted from the source hypothesis. We mitigate the impact of erroneous pseudo-labels that may arise due to a high ideal joint hypothesis risk by employing adaptive adversarial perturbation on the input of the target model. Our proposed source-free framework effectively addresses the challenging distribution shift scenarios and outperforms existing methods requiring source data and labels for training.