Optimal Projections for Classification with Naive Bayes

TL;DR

This paper introduces a projection pursuit method to optimize the basis for Naive Bayes classification, improving its discriminatory power, reducing dimensions, and enhancing visualization, with strong empirical results on benchmark datasets.

Contribution

It proposes a novel projection pursuit approach to find optimal bases for Naive Bayes, outperforming existing probabilistic models and rivaling SVMs in classification accuracy.

Findings

Outperforms popular probabilistic discriminant models

Highly competitive with Support Vector Machines

Effective in dimension reduction and visualization

Abstract

In the Naive Bayes classification model the class conditional densities are estimated as the products of their marginal densities along the cardinal basis directions. We study the problem of obtaining an alternative basis for this factorisation with the objective of enhancing the discriminatory power of the associated classification model. We formulate the problem as a projection pursuit to find the optimal linear projection on which to perform classification. Optimality is determined based on the multinomial likelihood within which probabilities are estimated using the Naive Bayes factorisation of the projected data. Projection pursuit offers the added benefits of dimension reduction and visualisation. We discuss an intuitive connection with class conditional independent components analysis, and show how this is realised visually in practical applications. The performance of the resulting classification models is investigated using a large collection of (162) publicly available benchmark data sets and in comparison with relevant alternatives. We find that the proposed approach substantially outperforms other popular probabilistic discriminant analysis models and is highly competitive with Support Vector Machines. Code to implement the proposed approach, in the form of an R package, is available from https://github.com/DavidHofmeyr/OPNB