Amazing Things Come From Having Many Good Models

TL;DR

This paper discusses the Rashomon Effect in machine learning, highlighting how multiple equally good models influence model simplicity, fairness, uncertainty, and policy, especially for noisy tabular data.

Contribution

It proposes a new perspective on ML that leverages the Rashomon Effect to improve model selection, fairness, interpretability, and policy implications.

Findings

Many good models exist for the same dataset.

The Rashomon Effect impacts fairness, interpretability, and uncertainty.

Guidelines for algorithm choice and policy considerations are discussed.

Abstract

The Rashomon Effect, coined by Leo Breiman, describes the phenomenon that there exist many equally good predictive models for the same dataset. This phenomenon happens for many real datasets and when it does, it sparks both magic and consternation, but mostly magic. In light of the Rashomon Effect, this perspective piece proposes reshaping the way we think about machine learning, particularly for tabular data problems in the nondeterministic (noisy) setting. We address how the Rashomon Effect impacts (1) the existence of simple-yet-accurate models, (2) flexibility to address user preferences, such as fairness and monotonicity, without losing performance, (3) uncertainty in predictions, fairness, and explanations, (4) reliable variable importance, (5) algorithm choice, specifically, providing advanced knowledge of which algorithms might be suitable for a given problem, and (6) public policy. We also discuss a theory of when the Rashomon Effect occurs and why. Our goal is to illustrate how the Rashomon Effect can have a massive impact on the use of machine learning for complex problems in society.