Random projections for Bayesian regression

TL;DR

This paper demonstrates that random projections can efficiently reduce data size in Bayesian linear regression while approximately preserving the posterior distribution, leading to faster computations with minimal loss of accuracy.

Contribution

It provides theoretical guarantees for data reduction via random projections in Bayesian regression and empirically validates the method's effectiveness.

Findings

Posterior distribution is approximated within a small error using reduced data.

The method significantly decreases computational time in Bayesian regression.

Theoretical conditions ensure the preservation of the entire distribution under random projections.

Abstract

This article deals with random projections applied as a data reduction technique for Bayesian regression analysis. We show sufficient conditions under which the entire $d$-dimensional distribution is approximately preserved under random projections by reducing the number of data points from $n$ to $k\in O(\operatorname{poly}(d/\varepsilon))$ in the case $n\gg d$. Under mild assumptions, we prove that evaluating a Gaussian likelihood function based on the projected data instead of the original data yields a $(1+O(\varepsilon))$-approximation in terms of the $\ell_2$ Wasserstein distance. Our main result shows that the posterior distribution of Bayesian linear regression is approximated up to a small error depending on only an $\varepsilon$-fraction of its defining parameters. This holds when using arbitrary Gaussian priors or the degenerate case of uniform distributions over $\mathbb{R}^d$ for $\beta$. Our empirical evaluations involve different simulated settings of Bayesian linear regression. Our experiments underline that the proposed method is able to recover the regression model up to small error while considerably reducing the total running time.