ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimation

TL;DR

ALSA introduces a novel logit-space based framework for more accurate out-of-distribution accuracy estimation, leveraging anchor points to capture subtle variations in model logits and outperform existing methods across diverse benchmarks.

Contribution

The paper proposes ALSA, a new method operating directly in logit space with learnable anchors, improving accuracy estimation under distribution shifts.

Findings

ALSA outperforms softmax- and similarity-based baselines.

ALSA maintains robustness under significant distribution shifts.

ALSA demonstrates effectiveness across vision, language, and graph benchmarks.

Abstract

Estimating model accuracy on unseen, unlabeled datasets is crucial for real-world machine learning applications, especially under distribution shifts that can degrade performance. Existing methods often rely on predicted class probabilities (softmax scores) or data similarity metrics. While softmax-based approaches benefit from representing predictions on the standard simplex, compressing logits into probabilities leads to information loss. Meanwhile, similarity-based methods can be computationally expensive and domain-specific, limiting their broader applicability. In this paper, we introduce ALSA (Anchors in Logit Space for Accuracy estimation), a novel framework that preserves richer information by operating directly in the logit space. Building on theoretical insights and empirical observations, we demonstrate that the aggregation and distribution of logits exhibit a strong correlation with the predictive performance of the model. To exploit this property, ALSA employs an anchor-based modeling strategy: multiple learnable anchors are initialized in logit space, each assigned an influence function that captures subtle variations in the logits. This allows ALSA to provide robust and accurate performance estimates across a wide range of distribution shifts. Extensive experiments on vision, language, and graph benchmarks demonstrate ALSA's superiority over both softmax- and similarity-based baselines. Notably, ALSA's robustness under significant distribution shifts highlights its potential as a practical tool for reliable model evaluation.