Estimation of Confidence Bounds in Binary Classification using Wilson Score Kernel Density Estimation

TL;DR

This paper introduces Wilson Score Kernel Density Classification, a new kernel-based method for estimating confidence bounds in binary classifiers, enhancing reliability in critical tasks with lower computational costs.

Contribution

The paper presents a novel Wilson Score Kernel Density Estimator for confidence bounds in binary classification, applicable to various feature extractors and more efficient than Gaussian Process Classification.

Findings

Performs comparably to Gaussian Process Classification

Effective in selective classification across multiple datasets

Lower computational complexity than existing methods

Abstract

The performance and ease of use of deep learning-based binary classifiers have improved significantly in recent years. This has opened up the potential for automating critical inspection tasks, which have traditionally only been trusted to be done manually. However, the application of binary classifiers in critical operations depends on the estimation of reliable confidence bounds such that system performance can be ensured up to a given statistical significance. We present Wilson Score Kernel Density Classification, which is a novel kernel-based method for estimating confidence bounds in binary classification. The core of our method is the Wilson Score Kernel Density Estimator, which is a function estimator for estimating confidence bounds in Binomial experiments with conditionally varying success probabilities. Our method is evaluated in the context of selective classification on four different datasets, illustrating its use as a classification head of any feature extractor, including vision foundation models. Our proposed method shows similar performance to Gaussian Process Classification, but at a lower computational complexity.