A Flexible Empirical Bayes Approach to Generalized Linear Models, with Applications to Sparse Logistic Regression

TL;DR

This paper presents a novel, tuning-free empirical Bayes variational inference method for Bayesian generalized linear models, offering a scalable, unified framework that improves sparse logistic regression performance.

Contribution

It introduces a flexible, mean-field variational inference approach that automatically estimates priors and is applicable to various exponential family distributions.

Findings

Superior predictive performance in sparse logistic regression

Scalable algorithms like L-BFGS and stochastic gradient descent used

Unified framework applicable to diverse likelihood and prior combinations

Abstract

We introduce a flexible empirical Bayes approach for fitting Bayesian generalized linear models. Specifically, we adopt a novel mean-field variational inference (VI) method and the prior is estimated within the VI algorithm, making the method tuning-free. Unlike traditional VI methods that optimize the posterior density function, our approach directly optimizes the posterior mean and prior parameters. This formulation reduces the number of parameters to optimize and enables the use of scalable algorithms such as L-BFGS and stochastic gradient descent. Furthermore, our method automatically determines the optimal posterior based on the prior and likelihood, distinguishing it from existing VI methods that often assume a Gaussian variational. Our approach represents a unified framework applicable to a wide range of exponential family distributions, removing the need to develop unique VI methods for each combination of likelihood and prior distributions. We apply the framework to solve sparse logistic regression and demonstrate the superior predictive performance of our method in extensive numerical studies, by comparing it to prevalent sparse logistic regression approaches.