Reconciling Predictive Multiplicity in Practice

TL;DR

This paper empirically evaluates the Reconcile algorithm for addressing model multiplicity in predictive probabilities, extends it to causal inference, and demonstrates its effectiveness across multiple datasets and settings.

Contribution

It provides the first extension of Reconcile to causal inference, analyzes its theoretical properties, and compares its performance with existing solutions.

Findings

Reconcile effectively reduces model disagreement on fairness datasets.

The extended Reconcile algorithm improves causal effect estimation accuracy.

Empirical results confirm Reconcile's practical applicability across domains.

Abstract

Many machine learning applications predict individual probabilities, such as the likelihood that a person develops a particular illness. Since these probabilities are unknown, a key question is how to address situations in which different models trained on the same dataset produce varying predictions for certain individuals. This issue is exemplified by the model multiplicity (MM) phenomenon, where a set of comparable models yield inconsistent predictions. Roth, Tolbert, and Weinstein recently introduced a reconciliation procedure, the Reconcile algorithm, to address this problem. Given two disagreeing models, the algorithm leverages their disagreement to falsify and improve at least one of the models. In this paper, we empirically analyze the Reconcile algorithm using five widely-used fairness datasets: COMPAS, Communities and Crime, Adult, Statlog (German Credit Data), and the ACS Dataset. We examine how Reconcile fits within the model multiplicity literature and compare it to existing MM solutions, demonstrating its effectiveness. We also discuss potential improvements to the Reconcile algorithm theoretically and practically. Finally, we extend the Reconcile algorithm to the setting of causal inference, given that different competing estimators can again disagree on specific causal average treatment effect (CATE) values. We present the first extension of the Reconcile algorithm in causal inference, analyze its theoretical properties, and conduct empirical tests. Our results confirm the practical effectiveness of Reconcile and its applicability across various domains.