Variational empirical Bayes variable selection in high-dimensional logistic regression

TL;DR

This paper introduces a novel variational approximation method for high-dimensional logistic regression variable selection, improving computational efficiency while maintaining strong theoretical guarantees.

Contribution

It develops a variational approximation directly for the model space in empirical Bayes logistic regression, enhancing computational efficiency and theoretical properties.

Findings

Strong performance in simulation studies

Inherits selection consistency from the posterior

Efficient approximation for high-dimensional data

Abstract

Logistic regression involving high-dimensional covariates is a practically important problem. Often the goal is variable selection, i.e., determining which few of the many covariates are associated with the binary response. Unfortunately, the usual Bayesian computations can be quite challenging and expensive. Here we start with a recently proposed empirical Bayes solution, with strong theoretical convergence properties, and develop a novel and computationally efficient variational approximation thereof. One such novelty is that we develop this approximation directly for the marginal distribution on the model space, rather than on the regression coefficients themselves. We demonstrate the method's strong performance in simulations, and prove that our variational approximation inherits the strong selection consistency property satisfied by the posterior distribution that it is approximating.