CASHomon Sets: Efficient Rashomon Sets Across Multiple Model Classes and their Hyperparameters

TL;DR

This paper introduces an efficient method for identifying nearly optimal model sets across multiple classes and hyperparameters, aiding model selection and interpretability in machine learning.

Contribution

It proposes TruVaRImp, a novel active learning algorithm for level set estimation in the CASH setting, with proven convergence guarantees and superior performance.

Findings

TruVaRImp reliably identifies CASHomon sets in synthetic and real datasets.

It outperforms naive sampling, Bayesian optimization, and classical methods.

Analysis reveals variability in feature importance across model classes.

Abstract

Rashomon sets are model sets within one model class that perform nearly as well as a reference model from the same model class. They reveal the existence of alternative well-performing models, which may support different interpretations. This enables selecting models that match domain knowledge, hidden constraints, or user preferences. However, efficient construction methods currently exist for only a few model classes. Applied machine learning usually searches many model classes, and the best class is unknown beforehand. We therefore study Rashomon sets in the combined algorithm selection and hyperparameter optimization (CASH) setting and call them CASHomon sets. We propose TruVaRImp, a model-based active learning algorithm for level set estimation with an implicit threshold, and provide convergence guarantees. On synthetic and real-world datasets, TruVaRImp reliably identifies CASHomon sets members and matches or outperforms naive sampling, Bayesian optimization, classical and implicit level set estimation methods, and other baselines. Our analyses of predictive multiplicity and feature-importance variability across model classes question the common practice of interpreting data through a single model class.