A Fast Linear Regression via SVD and Marginalization

TL;DR

This paper introduces a fast numerical method for Bayesian linear regression that leverages SVD and marginalization to efficiently compute posterior moments, reducing computational complexity especially for hierarchical models.

Contribution

The paper presents a novel analytical approach that transforms the problem into a lower-dimensional integral, enabling faster computation of posterior moments in Bayesian linear regression.

Findings

Efficient computation of posterior moments using SVD-based change of basis.

Reduction of high-dimensional integrals to lower-dimensional ones for faster evaluation.

Numerical demonstrations showing the method's effectiveness and generalization to hierarchical models.

Abstract

We describe a numerical scheme for evaluating the posterior moments of Bayesian linear regression models with partial pooling of the coefficients. The principal analytical tool of the evaluation is a change of basis from coefficient space to the space of singular vectors of the matrix of predictors. After this change of basis and an analytical integration, we reduce the problem of finding moments of a density over k + m dimensions, to finding moments of an m-dimensional density, where k is the number of coefficients and k + m is the dimension of the posterior. Moments can then be computed using, for example, MCMC, the trapezoid rule, or adaptive Gaussian quadrature. An evaluation of the SVD of the matrix of predictors is the dominant computational cost and is performed once during the precomputation stage. We demonstrate numerical results of the algorithm. The scheme described in this paper generalizes naturally to multilevel and multi-group hierarchical regression models where normal-normal parameters appear.