Risk-based Calibration for Generative Classifiers

TL;DR

This paper introduces risk-based calibration (RC), a novel iterative learning method for generative classifiers that directly optimizes for supervised classification metrics, significantly improving both training and generalization errors.

Contribution

The paper presents a new RC procedure that refines generative classifiers by aligning their training process with 0-1 loss, bridging the gap between traditional generative methods and supervised learning.

Findings

RC outperforms traditional methods on 20 datasets

Significant reduction in training and generalization errors

Applicable to naive Bayes and quadratic discriminant analysis

Abstract

Generative classifiers are constructed on the basis of a joint probability distribution and are typically learned using closed-form procedures that rely on data statistics and maximize scores related to data fitting. However, these scores are not directly linked to supervised classification metrics such as the error, i.e., the expected 0-1 loss. To address this limitation, we propose a learning procedure called risk-based calibration (RC) that iteratively refines the generative classifier by adjusting its joint probability distribution according to the 0-1 loss in training samples. This is achieved by reinforcing data statistics associated with the true classes while weakening those of incorrect classes. As a result, the classifier progressively assigns higher probability to the correct labels, improving its training error. Results on 20 heterogeneous datasets using both na\"ive Bayes and quadratic discriminant analysis show that RC significantly outperforms closed-form learning procedures in terms of both training error and generalization error. In this way, RC bridges the gap between traditional generative approaches and learning procedures guided by performance measures, ensuring a closer alignment with supervised classification objectives.