A Theoretical Analysis of Logistic Regression and Bayesian Classifiers

TL;DR

This paper provides a theoretical comparison between logistic regression and Bayesian classifiers, highlighting their differences, conditions for equivalence, and implications for model specification in statistical classification.

Contribution

It offers a formal analysis of the fundamental differences and conditions under which logistic regression and Bayesian classifiers produce similar or different results.

Findings

Logistic regression is a less general form of Bayesian classifier.

Correct class distribution specification is essential for logistic regression.

In certain cases, both classifiers yield identical predicted probabilities.

Abstract

This study aims to show the fundamental difference between logistic regression and Bayesian classifiers in the case of exponential and unexponential families of distributions, yielding the following findings. First, the logistic regression is a less general representation of a Bayesian classifier. Second, one should suppose distributions of classes for the correct specification of logistic regression equations. Third, in specific cases, there is no difference between predicted probabilities from correctly specified generative Bayesian classifier and discriminative logistic regression.