Uncertainty Quantification for Rule-Based Models

TL;DR

This paper introduces a novel uncertainty quantification framework for rule-based models, enabling confidence estimation and abstaining classification in out-of-distribution scenarios, addressing a gap in existing methods.

Contribution

It proposes a meta-model that estimates prediction accuracy and confidence for any binary classifier, including rule-based models, without requiring probabilistic outputs.

Findings

Effective confidence estimation for rule-based models

Improved abstaining classifier performance in OOD scenarios

Versatile framework applicable to various binary classifiers

Abstract

Rule-based classification models described in the language of logic directly predict boolean values, rather than modeling a probability and translating it into a prediction as done in statistical models. The vast majority of existing uncertainty quantification approaches rely on models providing continuous output not available to rule-based models. In this work, we propose an uncertainty quantification framework in the form of a meta-model that takes any binary classifier with binary output as a black box and estimates the prediction accuracy of that base model at a given input along with a level of confidence on that estimation. The confidence is based on how well that input region is explored and is designed to work in any OOD scenario. We demonstrate the usefulness of this uncertainty model by building an abstaining classifier powered by it and observing its performance in various scenarios.