Credal Model Averaging for classification: representing prior ignorance and expert opinions

TL;DR

This paper introduces Credal Model Averaging (CMA), a Bayesian approach that manages model uncertainty by using sets of priors, automates sensitivity analysis, and improves classification robustness, especially on small datasets.

Contribution

It proposes CMA as a novel extension of BMA that handles prior ignorance and domain knowledge, providing a systematic way to detect prior-dependent instances and improve classification.

Findings

CMA effectively detects prior-dependent instances.

BMA performs poorly on prior-dependent instances identified by CMA.

CMA enhances model robustness and sensitivity analysis.

Abstract

Bayesian model averaging (BMA) is the state of the art approach for overcoming model uncertainty. Yet, especially on small data sets, the results yielded by BMA might be sensitive to the prior over the models. Credal Model Averaging (CMA) addresses this problem by substituting the single prior over the models by a set of priors (credal set). Such approach solves the problem of how to choose the prior over the models and automates sensitivity analysis. We discuss various CMA algorithms for building an ensemble of logistic regressors characterized by different sets of covariates. We show how CMA can be appropriately tuned to the case in which one is prior-ignorant and to the case in which instead domain knowledge is available. CMA detects prior-dependent instances, namely instances in which a different class is more probable depending on the prior over the models. On such instances CMA suspends the judgment, returning multiple classes. We thoroughly compare different BMA and CMA variants on a real case study, predicting presence of Alpine marmot burrows in an Alpine valley. We find that BMA is almost a random guesser on the instances recognized as prior-dependent by CMA.