Confusion Matrices and Accuracy Statistics for Binary Classifiers Using Unlabeled Data: The Diagnostic Test Approach

TL;DR

This paper adapts diagnostic test methods to estimate confusion matrices and accuracy for binary classifiers on unlabeled data, enabling evaluation without labeled datasets.

Contribution

It introduces a novel approach to estimate confusion matrices and accuracy statistics for classifiers using unlabeled data, extending diagnostic test techniques.

Findings

Method successfully estimates confusion matrices without labeled data

Applicable to both supervised and unsupervised classifiers

Provides a new tool for classifier evaluation in unlabeled settings

Abstract

Medical researchers have solved the problem of estimating the sensitivity and specificity of binary medical diagnostic tests without gold standard tests for comparison. That problem is the same as estimating confusion matrices for classifiers on unlabeled data. This article describes how to modify the diagnostic test solutions to estimate confusion matrices and accuracy statistics for supervised or unsupervised binary classifiers on unlabeled data.