Semi-supervised linear regression with missing covariates

TL;DR

This paper develops new methods for linear regression with missing covariates, leveraging both labeled and unlabeled data, and provides theoretical guarantees showing their optimality under various missingness patterns.

Contribution

It introduces estimators for regression with missing covariates that are rate optimal and applicable to both sparse and non-sparse settings, filling a gap in the theoretical understanding.

Findings

Proposed estimators achieve minimax optimal risk bounds.

Established the first matching bounds for missing data in supervised settings.

Demonstrated effectiveness through simulations and real data application.

Abstract

Missing values in datasets are common in applied statistics. For regression problems, theoretical work thus far has largely considered the issue of missing covariates as distinct from missing responses. However, in practice, many datasets have both forms of missingness. Motivated by this gap, we study linear regression with a labelled dataset containing missing covariates, potentially alongside an unlabelled dataset. We consider both structured (blockwise-missing) and unstructured missingness patterns, along with sparse and non-sparse regression parameters. For the non-sparse case, we provide an estimator based on imputing the missing data combined with a reweighting step. For the high-dimensional sparse case, we use a modified version of the Dantzig selector. We provide non-asymptotic upper bounds on the risk of both procedures. These are matched by several new minimax lower bounds, demonstrating the rate optimality of our estimators. Notably, even when the linear model is well-specified, our results characterise substantial differences in the minimax rates when unlabelled data is present relative to the fully supervised setting. Particular consequences of our sparse and non-sparse results include the first matching upper and lower bounds on the minimax rate for the supervised setting when either unstructured or structured missingness is present. Our theory is coupled with extensive simulations and a semi-synthetic application to the California housing dataset.