Can we Agree? On the Rash\=omon Effect and the Reliability of Post-Hoc Explainable AI

TL;DR

This paper investigates how the reliability of post-hoc explanations for machine learning models improves with increased sample size, highlighting the importance of data quantity for trustworthy interpretability.

Contribution

It demonstrates the impact of sample size on explanation stability and agreement, providing empirical guidance for reliable post-hoc interpretability in ML models.

Findings

Explanations converge as sample size increases.

High variability in explanations with fewer than 128 samples.

Ensemble methods improve explanation agreement.

Abstract

The Rash\=omon effect poses challenges for deriving reliable knowledge from machine learning models. This study examined the influence of sample size on explanations from models in a Rash\=omon set using SHAP. Experiments on 5 public datasets showed that explanations gradually converged as the sample size increased. Explanations from <128 samples exhibited high variability, limiting reliable knowledge extraction. However, agreement between models improved with more data, allowing for consensus. Bagging ensembles often had higher agreement. The results provide guidance on sufficient data to trust explanations. Variability at low samples suggests that conclusions may be unreliable without validation. Further work is needed with more model types, data domains, and explanation methods. Testing convergence in neural networks and with model-specific explanation methods would be impactful. The approaches explored here point towards principled techniques for eliciting knowledge from ambiguous models.