Deriving discriminative classifiers from generative models

TL;DR

This paper demonstrates that classifiers traditionally considered generative can also be derived discriminatively from the same models, blurring the distinction between generative and discriminative approaches.

Contribution

It provides a general theoretical framework for deriving discriminative classifiers from generative models, including new extensions of Naive Bayes and Hidden Markov Chains.

Findings

Generative classifiers can be computed discriminatively from the same model.

Theoretical unification of Naive Bayes and Hidden Markov Chain classifiers.

Application of the approach to NLP tasks shows practical benefits.

Abstract

We deal with Bayesian generative and discriminative classifiers. Given a model distribution $p(x, y)$, with the observation $y$ and the target $x$, one computes generative classifiers by firstly considering $p(x, y)$ and then using the Bayes rule to calculate $p(x | y)$. A discriminative model is directly given by $p(x | y)$, which is used to compute discriminative classifiers. However, recent works showed that the Bayesian Maximum Posterior classifier defined from the Naive Bayes (NB) or Hidden Markov Chain (HMC), both generative models, can also match the discriminative classifier definition. Thus, there are situations in which dividing classifiers into "generative" and "discriminative" is somewhat misleading. Indeed, such a distinction is rather related to the way of computing classifiers, not to the classifiers themselves. We present a general theoretical result specifying how a generative classifier induced from a generative model can also be computed in a discriminative way from the same model. Examples of NB and HMC are found again as particular cases, and we apply the general result to two original extensions of NB, and two extensions of HMC, one of which being original. Finally, we shortly illustrate the interest of the new discriminative way of computing classifiers in the Natural Language Processing (NLP) framework.