Why are there many equally good models? An Anatomy of the Rashomon Effect

TL;DR

This paper investigates the Rashomon effect in machine learning, explaining why multiple models can perform equally well and categorizing the causes into statistical, structural, and procedural sources, with implications for various applications.

Contribution

It provides a unified framework for understanding the origins of model multiplicity, integrating insights from multiple disciplines and discussing its implications.

Findings

Statistical sources decrease with more data.

Structural sources persist asymptotically and require different data or assumptions.

Procedural sources depend on optimization choices and constraints.

Abstract

The Rashomon effect -- the existence of multiple, distinct models that achieve nearly equivalent predictive performance -- has emerged as a fundamental phenomenon in modern machine learning and statistics. In this paper, we explore the causes underlying the Rashomon effect, organizing them into three categories: statistical sources arising from finite samples and noise in the data-generating process; structural sources arising from non-convexity of optimization objectives and unobserved variables that create fundamental non-identifiability; and procedural sources arising from limitations of optimization algorithms and deliberate restrictions to suboptimal model classes. We synthesize insights from machine learning, statistics, and optimization literature to provide a unified framework for understanding why the multiplicity of good models arises. A key distinction emerges: statistical multiplicity diminishes with more data, structural multiplicity persists asymptotically and cannot be resolved without different data or additional assumptions, and procedural multiplicity reflects choices made by practitioners. Beyond characterizing causes, we discuss both the challenges and opportunities presented by the Rashomon effect, including implications for inference, interpretability, fairness, and decision-making under uncertainty.