Selective Probabilistic Classifier Based on Hypothesis Testing

TL;DR

This paper introduces a hypothesis testing-based rejection method for classifiers using probabilistic networks, effectively reducing false positives in safety-critical applications by estimating outcome distributions.

Contribution

It presents a novel rejection approach leveraging hypothesis testing with probabilistic networks to improve false positive control in classification tasks.

Findings

Achieves lower false positive ratios than existing methods.

Provides broader operational coverage in safety applications.

Outperforms Softmax Response in experiments on COCO and CIFAR datasets.

Abstract

In this paper, we propose a simple yet effective method to deal with the violation of the Closed-World Assumption for a classifier. Previous works tend to apply a threshold either on the classification scores or the loss function to reject the inputs that violate the assumption. However, these methods cannot achieve the low False Positive Ratio (FPR) required in safety applications. The proposed method is a rejection option based on hypothesis testing with probabilistic networks. With probabilistic networks, it is possible to estimate the distribution of outcomes instead of a single output. By utilizing Z-test over the mean and standard deviation for each class, the proposed method can estimate the statistical significance of the network certainty and reject uncertain outputs. The proposed method was experimented on with different configurations of the COCO and CIFAR datasets. The performance of the proposed method is compared with the Softmax Response, which is a known top-performing method. It is shown that the proposed method can achieve a broader range of operation and cover a lower FPR than the alternative.