Expectation Propagation for Neural Networks with Sparsity-promoting Priors

TL;DR

This paper introduces an expectation propagation method for nonlinear regression with neural networks that incorporate sparsity-promoting hierarchical priors, enabling efficient Bayesian inference and flexible model structures.

Contribution

It develops a computationally efficient EP-based algorithm for neural networks with hierarchical sparsity priors, extending flexibility in prior choices and model structures.

Findings

The approach performs well on simulated data.

It achieves competitive results on real-world datasets.

The method scales similarly to sparse linear models.

Abstract

We propose a novel approach for nonlinear regression using a two-layer neural network (NN) model structure with sparsity-favoring hierarchical priors on the network weights. We present an expectation propagation (EP) approach for approximate integration over the posterior distribution of the weights, the hierarchical scale parameters of the priors, and the residual scale. Using a factorized posterior approximation we derive a computationally efficient algorithm, whose complexity scales similarly to an ensemble of independent sparse linear models. The approach enables flexible definition of weight priors with different sparseness properties such as independent Laplace priors with a common scale parameter or Gaussian automatic relevance determination (ARD) priors with different relevance parameters for all inputs. The approach can be extended beyond standard activation functions and NN model structures to form flexible nonlinear predictors from multiple sparse linear models. The effects of the hierarchical priors and the predictive performance of the algorithm are assessed using both simulated and real-world data. Comparisons are made to two alternative models with ARD priors: a Gaussian process with a NN covariance function and marginal maximum a posteriori estimates of the relevance parameters, and a NN with Markov chain Monte Carlo integration over all the unknown model parameters.