On the Bayes Inconsistency of Disagreement Discrepancy Surrogates
Neil G. Marchant, Andrew C. Cullen, Feng Liu, Sarah M. Erfani

TL;DR
This paper reveals that existing surrogate losses for disagreement discrepancy in neural networks are Bayes inconsistent, introduces new theoretical bounds, and proposes a consistent alternative that improves robustness under distribution shifts.
Contribution
The paper proves the Bayes inconsistency of current surrogates, develops bounds on the optimality gap, and introduces a new consistent disagreement loss paired with cross-entropy.
Findings
New disagreement loss is more accurate in estimating disagreement discrepancy.
Proposed method improves robustness under adversarial distribution shifts.
Theoretical bounds guide the design of consistent surrogates.
Abstract
Deep neural networks often fail when deployed in real-world contexts due to distribution shift, a critical barrier to building safe and reliable systems. An emerging approach to address this problem relies on \emph{disagreement discrepancy} -- a measure of how the disagreement between two models changes under a shifting distribution. The process of maximizing this measure has seen applications in bounding error under shifts, testing for harmful shifts, and training more robust models. However, this optimization involves the non-differentiable zero-one loss, necessitating the use of practical surrogate losses. We prove that existing surrogates for disagreement discrepancy are not Bayes consistent, revealing a fundamental flaw: maximizing these surrogates can fail to maximize the true disagreement discrepancy. To address this, we introduce new theoretical results providing both upper and…
Peer Reviews
Decision·ICLR 2026 Poster
1. I think paper makes clear theoretical contribution, which is also important. I've seen the works on disagreement discrepancy before as applied for testing, and have found the application to be hacky. The current contribution studies the soundness of it, and indeed, the finding that prior approaches are not sound is original and of significance. 2. The paper is also thorough (I like the appendix, although I didn't verify everything too closely), but the decomposition of the risk using density
1. Overall, I do like the paper from the theoretical contribution perspective. However, I'm not sure I followed the experimental section that well. Could authors clarify what is max disagreement discrepancy (in Figure 1) and how it is defined and estimated? I also feel writing wise, the paper assumes too much background on prior work. 2. Furthermore, I think while the quality of surrogate is the main focus of the paper, the paper does not connect to downstream applications. I'd assume in testin
The paper tackles a fundamental open question in the theory of domain adaptation and covariate shift: whether the surrogate losses used for estimating disagreement discrepancy are theoretically sound. Few prior works examined the consistency properties of such surrogates; most focused on empirical performance. The authors provide what appears to be the first demonstration that the popular disagreement surrogates of Rosenfeld & Garg (2023) and Ginsberg et al. (2023) are Bayes-inconsistent in the
The main theoretical guarantee - Bayes consistency - is asymptotic and does not specify convergence rates or finite-sample behavior. This makes it unclear how the surrogate behaves with limited data or under model misspecification. Without sample-complexity or generalization bounds, practitioners lack quantitative guidance on when the theoretical improvement translates into measurable gains. The analysis assumes covariate shift only, i.e., $P(Y \textbar X) $ remains constant across domains. In
The paper is clear and address a perennial problem of understanding how covariate shift can affect classifiers. Overall, the paper is well written.
There are a few points that could be made more clear (questions below). Additionally, is it possible to use the consistent form as a bound on error to create distributional robustness? I understand the adversarial attack is a step in this direction, but commenting on how this connects to making better unsupervised domain adaption algorithms would be interesting.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Advanced Neural Network Applications · Explainable Artificial Intelligence (XAI)
