What are the Differences between Bayesian Classifiers and Mutual-Information Classifiers?

TL;DR

This paper compares Bayesian and mutual-information classifiers for binary classification, highlighting their theoretical differences, especially in handling class imbalance and reject options, with empirical examples illustrating their performance.

Contribution

It provides a formal analysis of the decision rules and parameter redundancy in Bayesian classifiers and demonstrates the advantages of mutual-information classifiers in imbalanced scenarios.

Findings

Mutual-information classifiers offer an objective solution balancing error and reject types.

Bayesian classifiers exhibit parameter redundancy and non-consistency in cost interpretation.

Mutual-information classifiers perform better in class-imbalanced cases.

Abstract

In this study, both Bayesian classifiers and mutual information classifiers are examined for binary classifications with or without a reject option. The general decision rules in terms of distinctions on error types and reject types are derived for Bayesian classifiers. A formal analysis is conducted to reveal the parameter redundancy of cost terms when abstaining classifications are enforced. The redundancy implies an intrinsic problem of "non-consistency" for interpreting cost terms. If no data is given to the cost terms, we demonstrate the weakness of Bayesian classifiers in class-imbalanced classifications. On the contrary, mutual-information classifiers are able to provide an objective solution from the given data, which shows a reasonable balance among error types and reject types. Numerical examples of using two types of classifiers are given for confirming the theoretical differences, including the extremely-class-imbalanced cases. Finally, we briefly summarize the Bayesian classifiers and mutual-information classifiers in terms of their application advantages, respectively.