Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation

TL;DR

This paper introduces a method using the nuclear norm to assess model performance under distribution shifts by jointly considering prediction confidence and dispersity, leading to more accurate accuracy estimation.

Contribution

It proposes a novel approach leveraging the nuclear norm to characterize both confidence and dispersity for unsupervised accuracy estimation under distribution shifts.

Findings

Nuclear norm outperforms existing methods in accuracy estimation.

Effective across various models, datasets, and distribution shifts.

Explores alternative measures and addresses limitations under class imbalance.

Abstract

This work aims to assess how well a model performs under distribution shifts without using labels. While recent methods study prediction confidence, this work reports prediction dispersity is another informative cue. Confidence reflects whether the individual prediction is certain; dispersity indicates how the overall predictions are distributed across all categories. Our key insight is that a well-performing model should give predictions with high confidence and high dispersity. That is, we need to consider both properties so as to make more accurate estimates. To this end, we use the nuclear norm that has been shown to be effective in characterizing both properties. Extensive experiments validate the effectiveness of nuclear norm for various models (e.g., ViT and ConvNeXt), different datasets (e.g., ImageNet and CUB-200), and diverse types of distribution shifts (e.g., style shift and reproduction shift). We show that the nuclear norm is more accurate and robust in accuracy estimation than existing methods. Furthermore, we validate the feasibility of other measurements (e.g., mutual information maximization) for characterizing dispersity and confidence. Lastly, we investigate the limitation of the nuclear norm, study its improved variant under severe class imbalance, and discuss potential directions.