Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation

TL;DR

This paper introduces a dropout-based framework for efficiently exploring the Rashomon set to estimate and mitigate predictive multiplicity in classification models, improving speed and effectiveness.

Contribution

It proposes a novel dropout technique to explore the Rashomon set, providing theoretical insights and empirical results that outperform existing methods in estimating predictive multiplicity.

Findings

Outperforms baselines in predictive multiplicity metric estimation

Achieves up to 5000x speedup in exploration

Enables effective mitigation of predictive multiplicity

Abstract

Predictive multiplicity refers to the phenomenon in which classification tasks may admit multiple competing models that achieve almost-equally-optimal performance, yet generate conflicting outputs for individual samples. This presents significant concerns, as it can potentially result in systemic exclusion, inexplicable discrimination, and unfairness in practical applications. Measuring and mitigating predictive multiplicity, however, is computationally challenging due to the need to explore all such almost-equally-optimal models, known as the Rashomon set, in potentially huge hypothesis spaces. To address this challenge, we propose a novel framework that utilizes dropout techniques for exploring models in the Rashomon set. We provide rigorous theoretical derivations to connect the dropout parameters to properties of the Rashomon set, and empirically evaluate our framework through extensive experimentation. Numerical results show that our technique consistently outperforms baselines in terms of the effectiveness of predictive multiplicity metric estimation, with runtime speedup up to $20\times \sim 5000\times$. With efficient Rashomon set exploration and metric estimation, mitigation of predictive multiplicity is then achieved through dropout ensemble and model selection.