Demystifying Disagreement-on-the-Line in High Dimensions

TL;DR

This paper provides a theoretical analysis of disagreement-on-the-line phenomenon in high-dimensional models, linking disagreement and error across domains, supported by experiments on multiple datasets.

Contribution

It develops a theoretical framework for understanding disagreement in high-dimensional regression and identifies conditions for the disagreement-on-the-line phenomenon.

Findings

Disagreement correlates with prediction error in high dimensions.

The disagreement-on-the-line phenomenon occurs under specific conditions.

Experimental results align with theoretical predictions across datasets.

Abstract

Evaluating the performance of machine learning models under distribution shift is challenging, especially when we only have unlabeled data from the shifted (target) domain, along with labeled data from the original (source) domain. Recent work suggests that the notion of disagreement, the degree to which two models trained with different randomness differ on the same input, is a key to tackle this problem. Experimentally, disagreement and prediction error have been shown to be strongly connected, which has been used to estimate model performance. Experiments have led to the discovery of the disagreement-on-the-line phenomenon, whereby the classification error under the target domain is often a linear function of the classification error under the source domain; and whenever this property holds, disagreement under the source and target domain follow the same linear relation. In this work, we develop a theoretical foundation for analyzing disagreement in high-dimensional random features regression; and study under what conditions the disagreement-on-the-line phenomenon occurs in our setting. Experiments on CIFAR-10-C, Tiny ImageNet-C, and Camelyon17 are consistent with our theory and support the universality of the theoretical findings.