Expectation Propagation for Approximate Inference: Free Probability Framework

TL;DR

This paper applies free probability theory to analyze expectation propagation (EP) for approximate inference in large-scale generalized linear models, addressing computational challenges and demonstrating its effectiveness on gene selection tasks.

Contribution

It introduces a free probability framework to analyze and improve the scalability of EP in high-dimensional settings with specific data matrix properties.

Findings

Free probability analysis reduces computational complexity of EP.

EP performs well on gene selection microarray data.

Theoretical insights enable scalable approximate inference.

Abstract

We study asymptotic properties of expectation propagation (EP) -- a method for approximate inference originally developed in the field of machine learning. Applied to generalized linear models, EP iteratively computes a multivariate Gaussian approximation to the exact posterior distribution. The computational complexity of the repeated update of covariance matrices severely limits the application of EP to large problem sizes. In this study, we present a rigorous analysis by means of free probability theory that allows us to overcome this computational bottleneck if specific data matrices in the problem fulfill certain properties of asymptotic freeness. We demonstrate the relevance of our approach on the gene selection problem of a microarray dataset.