Scalable Bayesian inference on high-dimensional multivariate linear regression

TL;DR

This paper introduces two scalable Bayesian methods for high-dimensional multivariate linear regression, effectively estimating coefficients and error precision matrices while addressing computational challenges and uncertainty quantification.

Contribution

It proposes an exact Bayesian approach with spike and slab priors and a scalable two-step method that ignores response dependencies, improving computational efficiency and uncertainty estimation.

Findings

The exact method has computational complexity comparable to existing approaches.

The two-step method demonstrates selection consistency and posterior convergence.

Both methods perform well on synthetic and real datasets.

Abstract

We consider jointly estimating the coefficient matrix and the error precision matrix in high-dimensional multivariate linear regression models. Bayesian methods in this context often face computational challenges, leading to previous approaches that either utilize a generalized likelihood without ensuring the positive definiteness of the precision matrix or rely on maximization algorithms targeting only the posterior mode, thus failing to address uncertainty. In this work, we propose two Bayesian methods: an exact method and an approximate two-step method. We first propose an exact method based on spike and slab priors for the coefficient matrix and DAG-Wishart prior for the error precision matrix, whose computational complexity is comparable to the state-of-the-art generalized likelihood-based Bayesian method. To further enhance scalability, a two-step approach is developed by ignoring the dependency structure among response variables. This method estimates the coefficient matrix first, followed by the calculation of the posterior of the error precision matrix based on the estimated errors. We validate the two-step method by demonstrating (i) selection consistency and posterior convergence rates for the coefficient matrix and (ii) selection consistency for the directed acyclic graph (DAG) of errors. We demonstrate the practical performance of proposed methods through synthetic and real data analysis.