Improved Estimators for Semi-supervised High-dimensional Regression Model

TL;DR

This paper introduces an unbiased, consistent, and asymptotically normal estimator for the conditional variance in high-dimensional semi-supervised regression, with methods to improve it using zero-estimators and covariate selection algorithms.

Contribution

It proposes a novel estimator for variance in high-dimensional semi-supervised regression and introduces covariate selection algorithms to optimize estimator improvement.

Findings

Estimator is unbiased, consistent, and asymptotically normal.

Adding zero-estimators can reduce variance without bias.

Simulation results demonstrate theoretical properties.

Abstract

We study a linear high-dimensional regression model in a semi-supervised setting, where for many observations only the vector of covariates $X$ is given with no response $Y$. We do not make any sparsity assumptions on the vector of coefficients, and aim at estimating $\mathrm{Var}(Y|X)$. We propose an estimator, which is unbiased, consistent, and asymptotically normal. This estimator can be improved by adding zero-estimators arising from the unlabelled data. Adding zero-estimators does not affect the bias and potentially can reduce variance. In order to achieve optimal improvement, many zero-estimators should be used, but this raises the problem of estimating many parameters. Therefore, we introduce covariate selection algorithms that identify which zero-estimators should be used in order to improve the above estimator. We further illustrate our approach for other estimators, and present an algorithm that improves estimation for any given variance estimator. Our theoretical results are demonstrated in a simulation study.