Accounting for Underspecification in Statistical Claims of Model Superiority

TL;DR

This paper emphasizes the importance of accounting for model underspecification and training variance when making statistical claims of superiority in medical imaging models, showing that small seed differences can significantly affect results.

Contribution

The paper extends a statistical framework to include underspecification as a variance component, highlighting its impact on model comparison validity.

Findings

Small seed variability (~1%) increases evidence needed for claims

Underspecification significantly affects performance comparisons

Explicit modeling of training variance is crucial for validation

Abstract

Machine learning methods are increasingly applied in medical imaging, yet many reported improvements lack statistical robustness: recent works have highlighted that small but significant performance gains are highly likely to be false positives. However, these analyses do not take \emph{underspecification} into account -- the fact that models achieving similar validation scores may behave differently on unseen data due to random initialization or training dynamics. Here, we extend a recent statistical framework modeling false outperformance claims to include underspecification as an additional variance component. Our simulations demonstrate that even modest seed variability ($\sim1\%$) substantially increases the evidence required to support superiority claims. Our findings underscore the need for explicit modeling of training variance when validating medical imaging systems.