Efficient Approximate Inference with Walsh-Hadamard Variational Inference

TL;DR

This paper introduces Walsh-Hadamard Variational Inference, a novel method that enhances the efficiency and expressiveness of variational inference for complex models by leveraging Walsh-Hadamard-based factorization.

Contribution

It proposes a new Walsh-Hadamard-based factorization strategy to improve variational inference for over-parameterized models, reducing parameters and computational cost.

Findings

Reduces model parameterization significantly.

Accelerates inference computations.

Increases expressiveness of approximate posteriors.

Abstract

Variational inference offers scalable and flexible tools to tackle intractable Bayesian inference of modern statistical models like Bayesian neural networks and Gaussian processes. For largely over-parameterized models, however, the over-regularization property of the variational objective makes the application of variational inference challenging. Inspired by the literature on kernel methods, and in particular on structured approximations of distributions of random matrices, this paper proposes Walsh-Hadamard Variational Inference, which uses Walsh-Hadamard-based factorization strategies to reduce model parameterization, accelerate computations, and increase the expressiveness of the approximate posterior beyond fully factorized ones.