How to Control the Error Rates of Binary Classifiers

TL;DR

This paper introduces a method to control specific error rates of binary classifiers by integrating statistical hypothesis testing, enabling users to set and limit false positive or false negative rates post-training.

Contribution

It presents a novel approach to transform existing classifiers into statistical tests for targeted error rate control, bridging classification and hypothesis testing.

Findings

Method to convert classifiers into statistical tests

Calculation of classification p-values for error control

Effective control of false positive/negative rates

Abstract

The traditional binary classification framework constructs classifiers which may have good accuracy, but whose false positive and false negative error rates are not under users' control. In many cases, one of the errors is more severe and only the classifiers with the corresponding rate lower than the predefined threshold are acceptable. In this study, we combine binary classification with statistical hypothesis testing to control the target error rate of already trained classifiers. In particular, we show how to turn binary classifiers into statistical tests, calculate the classification p-values, and use them to limit the target error rate.