For high-dimensional hierarchical models, consider exchangeability of effects across covariates instead of across datasets

TL;DR

This paper proposes a new hierarchical Bayesian model that exchanges effects across covariates rather than datasets, improving statistical performance in high-dimensional settings like genetics.

Contribution

It introduces an alternative hierarchical model and empirical Bayes estimator tailored for high-dimensional data where covariates outnumber datasets.

Findings

Outperforms classic models when covariates exceed datasets

Demonstrates improved accuracy in high-dimensional regression and classification

Provides theoretical and empirical validation of the approach

Abstract

Hierarchical Bayesian methods enable information sharing across multiple related regression problems. While standard practice is to model regression parameters (effects) as (1) exchangeable across datasets and (2) correlated to differing degrees across covariates, we show that this approach exhibits poor statistical performance when the number of covariates exceeds the number of datasets. For instance, in statistical genetics, we might regress dozens of traits (defining datasets) for thousands of individuals (responses) on up to millions of genetic variants (covariates). When an analyst has more covariates than datasets, we argue that it is often more natural to instead model effects as (1) exchangeable across covariates and (2) correlated to differing degrees across datasets. To this end, we propose a hierarchical model expressing our alternative perspective. We devise an empirical Bayes estimator for learning the degree of correlation between datasets. We develop theory that demonstrates that our method outperforms the classic approach when the number of covariates dominates the number of datasets, and corroborate this result empirically on several high-dimensional multiple regression and classification problems.