Noise Covariance Estimation in Multi-Task High-dimensional Linear Models

TL;DR

This paper introduces a new, efficient estimator for the noise covariance matrix in multi-task high-dimensional linear models, achieving optimal convergence rates and outperforming existing methods.

Contribution

It develops a novel bias-corrected estimator for noise covariance that converges at the parametric rate and is computationally feasible, improving over traditional methods.

Findings

Estimator achieves $n^{-1/2}$ convergence rate with Gaussian covariates.

Proposed method outperforms method-of-moments in simulations.

Provides estimates of generalization error for multi-task estimators.

Abstract

This paper studies the multi-task high-dimensional linear regression models where the noise among different tasks is correlated, in the moderately high dimensional regime where sample size $n$ and dimension $p$ are of the same order. Our goal is to estimate the covariance matrix of the noise random vectors, or equivalently the correlation of the noise variables on any pair of two tasks. Treating the regression coefficients as a nuisance parameter, we leverage the multi-task elastic-net and multi-task lasso estimators to estimate the nuisance. By precisely understanding the bias of the squared residual matrix and by correcting this bias, we develop a novel estimator of the noise covariance that converges in Frobenius norm at the rate $n^{-1/2}$ when the covariates are Gaussian. This novel estimator is efficiently computable. Under suitable conditions, the proposed estimator of the noise covariance attains the same rate of convergence as the "oracle" estimator that knows in advance the regression coefficients of the multi-task model. The Frobenius error bounds obtained in this paper also illustrate the advantage of this new estimator compared to a method-of-moments estimator that does not attempt to estimate the nuisance. As a byproduct of our techniques, we obtain an estimate of the generalization error of the multi-task elastic-net and multi-task lasso estimators. Extensive simulation studies are carried out to illustrate the numerical performance of the proposed method.