Bayesian feature selection with strongly-regularizing priors maps to the Ising Model

TL;DR

This paper reveals that Bayesian feature selection with strongly-regularizing priors can be mapped to an Ising model, providing a universal and computationally efficient approach for high-dimensional feature selection.

Contribution

It establishes a universal connection between Bayesian feature selection and the Ising model, deriving explicit formulas for generalized linear models.

Findings

Feature selection reduces to computing Ising model magnetizations.

Derived explicit formulas for logistic and linear regression.

Applied method successfully to notMNIST dataset classification.

Abstract

Identifying small subsets of features that are relevant for prediction and/or classification tasks is a central problem in machine learning and statistics. The feature selection task is especially important, and computationally difficult, for modern datasets where the number of features can be comparable to, or even exceed, the number of samples. Here, we show that feature selection with Bayesian inference takes a universal form and reduces to calculating the magnetizations of an Ising model, under some mild conditions. Our results exploit the observation that the evidence takes a universal form for strongly-regularizing priors --- priors that have a large effect on the posterior probability even in the infinite data limit. We derive explicit expressions for feature selection for generalized linear models, a large class of statistical techniques that include linear and logistic regression. We illustrate the power of our approach by analyzing feature selection in a logistic regression-based classifier trained to distinguish between the letters B and D in the notMNIST dataset.