Bayesian Variable Selection in Multivariate Regression Under Collinearity in the Design Matrix

TL;DR

This paper investigates Bayesian multivariate regression with correlated errors, revealing that modeling such correlations can worsen estimation in low-information scenarios, and proposes a two-step estimation approach as a practical solution.

Contribution

It demonstrates that ignoring error correlation in Bayesian multivariate regression can improve estimation accuracy, especially in low-information settings, and suggests a simple two-step estimation method.

Findings

Modeling error correlation can worsen estimation in certain scenarios.

A two-step estimation procedure improves accuracy in low-information settings.

Using a diagonal covariance matrix can be beneficial despite potential correlations.

Abstract

We consider the problem of variable selection in Bayesian multivariate linear regression models, involving multiple response and predictor variables, under multivariate normal errors. In the absence of a known covariance structure, specifying a model with a non-diagonal covariance matrix is appealing. Modeling dependency in the random errors through a non-diagonal covariance matrix is generally expected to lead to improved estimation of the regression coefficients. In this article, we highlight an interesting exception: modeling the dependency in errors can significantly worsen both estimation and prediction. We demonstrate that Bayesian multi-outcome regression models using several popular variable selection priors can suffer from poor estimation properties in low-information settings--such as scenarios with weak signals, high correlation among predictors and responses, and small sample sizes. In such cases, the simultaneous estimation of all unknown parameters in the model becomes difficult when using a non-diagonal covariance matrix. Through simulation studies and a dataset with measurements from NIR spectroscopy, we illustrate that a two-step procedure--estimating the mean and the covariance matrix separately--can provide more accurate estimates in such cases. Thus, a potential solution to avoid the problem altogether is to routinely perform an additional analysis with a diagonal covariance matrix, even if the errors are expected to be correlated.