Naive Bayes Classifiers and One-hot Encoding of Categorical Variables

TL;DR

This paper examines the impact of incorrectly encoding categorical variables as one-hot vectors in Naive Bayes classifiers, analyzing the mathematical differences and experimental outcomes of the product-of-Bernoullis assumption versus the true categorical model.

Contribution

It provides a mathematical and experimental comparison between the product-of-Bernoullis assumption and the correct categorical Naive Bayes classifier when using one-hot encoding.

Findings

Classifiers often agree on the MAP label despite encoding differences.

Posterior probabilities tend to be higher under the PoB assumption.

Differences are analyzed mathematically and through experiments.

Abstract

This paper investigates the consequences of encoding a $K$-valued categorical variable incorrectly as $K$ bits via one-hot encoding, when using a Na\"{\i}ve Bayes classifier. This gives rise to a product-of-Bernoullis (PoB) assumption, rather than the correct categorical Na\"{\i}ve Bayes classifier. The differences between the two classifiers are analysed mathematically and experimentally. In our experiments using probability vectors drawn from a Dirichlet distribution, the two classifiers are found to agree on the maximum a posteriori class label for most cases, although the posterior probabilities are usually greater for the PoB case.