An Empirical Bayes Approach for High Dimensional Classification

TL;DR

This paper introduces an empirical Bayes method using Dirichlet process mixtures for high-dimensional classification, providing theoretical error bounds and an efficient variational Bayes algorithm suitable for ultra-high dimensions.

Contribution

It presents a novel empirical Bayes estimator for sparse mean differences and establishes theoretical links between estimation and classification errors.

Findings

The method achieves competitive classification accuracy.

Theoretical conditions for optimal and sub-optimal classifiers are provided.

An efficient parallelizable algorithm is developed for ultra-high dimensional data.

Abstract

We propose an empirical Bayes estimator based on Dirichlet process mixture model for estimating the sparse normalized mean difference, which could be directly applied to the high dimensional linear classification. In theory, we build a bridge to connect the estimation error of the mean difference and the misclassification error, also provide sufficient conditions of sub-optimal classifiers and optimal classifiers. In implementation, a variational Bayes algorithm is developed to compute the posterior efficiently and could be parallelized to deal with the ultra-high dimensional case.