On the Importance of Feature Separability in Predicting Out-Of-Distribution Error

TL;DR

This paper investigates how feature separability impacts out-of-distribution error prediction, proposing a new dataset-level score based on feature dispersion that correlates with model accuracy under distribution shifts.

Contribution

It introduces a novel feature dispersion-based score for estimating OOD accuracy, emphasizing the importance of inter-class dispersion over intra-class compactness.

Findings

Inter-class dispersion correlates strongly with accuracy.

Intra-class compactness does not reflect OOD performance.

The proposed method outperforms existing approaches in prediction and efficiency.

Abstract

Estimating the generalization performance is practically challenging on out-of-distribution (OOD) data without ground-truth labels. While previous methods emphasize the connection between distribution difference and OOD accuracy, we show that a large domain gap not necessarily leads to a low test accuracy. In this paper, we investigate this problem from the perspective of feature separability empirically and theoretically. Specifically, we propose a dataset-level score based upon feature dispersion to estimate the test accuracy under distribution shift. Our method is inspired by desirable properties of features in representation learning: high inter-class dispersion and high intra-class compactness. Our analysis shows that inter-class dispersion is strongly correlated with the model accuracy, while intra-class compactness does not reflect the generalization performance on OOD data. Extensive experiments demonstrate the superiority of our method in both prediction performance and computational efficiency.