On identification in ill-posed linear regression

TL;DR

This paper introduces a distribution-free framework for identifying parameters in ill-posed linear regression models, providing verifiable conditions and error bounds for linear dimensionality reduction algorithms, with improved convergence rates under certain conditions.

Contribution

It formalizes identifiability in ill-posed linear regression, defines a new identifiable parameter, and establishes verifiable conditions and sharp error bounds for relevant algorithms.

Findings

Algorithms achieve improved convergence rates with heavy-tailed features.

Verifiable conditions enable broad class of algorithms to estimate identifiable parameters.

Framework extends to nonlinear response-feature models.

Abstract

A novel framework is introduced to formalize identifiability in well-specified but ill-posed linear regression models. The framework is distribution-free and accommodates highly correlated features that may or may not relate to the response, reflecting typical real-data structures. First, the identifiable parameter is defined as the least-squares solution obtained by regressing the response on the largest subset of relevant features whose condition number does not exceed a specified threshold, and the relative risk incurred by using this predictor instead of the optimal one is quantified. Second, simple, verifiable conditions are provided under which a broad class of linear dimensionality reduction algorithms can estimate identifiable parameters; algorithms satisfying these conditions are termed statistically interpretable. Third, sharp high-probability error bounds are derived for these algorithms, with rates explicitly reflecting the degree of ill-posedness. With heavy-tailed features and sufficiently low effective rank, these algorithms achieve convergence rates that improve upon both the minimax least-squares rate and lower bounds for sparse estimation under sub-Gaussian features. Results are illustrated via simulations and a real-data application, in which effective rank grows logarithmically with dimension. The framework may extend to algorithms modeling nonlinear response-feature dependence.