EigenNet: A Bayesian hybrid of generative and conditional models for sparse learning

TL;DR

EigenNet is a Bayesian hybrid model that leverages data eigenstructures for improved variable selection in high-dimensional settings, outperforming traditional methods like lasso and elastic net especially with limited training data.

Contribution

The paper introduces EigenNet, a novel Bayesian hybrid model that integrates generative and conditional models using eigenspace reparameterization for efficient variable selection.

Findings

EigenNet outperforms lasso, elastic net, and Bayesian lasso in prediction accuracy.

It effectively utilizes correlation information in data for variable selection.

The model is computationally efficient due to eigenspace reparameterization.

Abstract

It is a challenging task to select correlated variables in a high dimensional space. To address this challenge, the elastic net has been developed and successfully applied to many applications. Despite its great success, the elastic net does not explicitly use correlation information embedded in data to select correlated variables. To overcome this limitation, we present a novel Bayesian hybrid model, the EigenNet, that uses the eigenstructures of data to guide variable selection. Specifically, it integrates a sparse conditional classification model with a generative model capturing variable correlations in a principled Bayesian framework. We reparameterize the hybrid model in the eigenspace to avoid overfiting and to increase the computational efficiency of its MCMC sampler. Furthermore, we provide an alternative view to the EigenNet from a regularization perspective: the EigenNet has an adaptive eigenspace-based composite regularizer, which naturally generalizes the $l_{1/2}$ regularizer used by the elastic net. Experiments on synthetic and real data show that the EigenNet significantly outperforms the lasso, the elastic net, and the Bayesian lasso in terms of prediction accuracy, especially when the number of training samples is smaller than the number of variables.