Bridging the Gap: Unifying the Training and Evaluation of Neural Network Binary Classifiers

TL;DR

This paper introduces a unified training method for neural network binary classifiers that directly optimizes evaluation metrics like F1-Score by combining differentiable approximations and probabilistic confusion matrix modeling.

Contribution

It presents a novel approach that unifies training and evaluation by enabling direct optimization of metrics, overcoming limitations of existing techniques.

Findings

Effective in optimizing F1-Score and accuracy

Outperforms traditional training methods in experiments

Applicable across multiple domains

Abstract

While neural network binary classifiers are often evaluated on metrics such as Accuracy and $F_1$-Score, they are commonly trained with a cross-entropy objective. How can this training-evaluation gap be addressed? While specific techniques have been adopted to optimize certain confusion matrix based metrics, it is challenging or impossible in some cases to generalize the techniques to other metrics. Adversarial learning approaches have also been proposed to optimize networks via confusion matrix based metrics, but they tend to be much slower than common training methods. In this work, we propose a unifying approach to training neural network binary classifiers that combines a differentiable approximation of the Heaviside function with a probabilistic view of the typical confusion matrix values using soft sets. Our theoretical analysis shows the benefit of using our method to optimize for a given evaluation metric, such as $F_1$-Score, with soft sets, and our extensive experiments show the effectiveness of our approach in several domains.