A Projector-Based Approach to Quantifying Total and Excess Uncertainties for Sketched Linear Regression

TL;DR

This paper introduces a projector-based method to precisely analyze how sketching techniques impact the statistical properties of linear regression solutions, enabling better understanding and diagnostics of uncertainties introduced by sketching.

Contribution

It presents an exact, minimal-assumption projector-based framework for analyzing sketched linear regression, allowing for precise quantification of uncertainties and bias-variance trade-offs.

Findings

Quantifies excess uncertainties in sketched regression

Provides diagnostics for sketching scheme design

Enables exact downstream analysis regardless of sketching method

Abstract

Linear regression is a classic method of data analysis. In recent years, sketching -- a method of dimension reduction using random sampling, random projections, or both -- has gained popularity as an effective computational approximation when the number of observations greatly exceeds the number of variables. In this paper, we address the following question: How does sketching affect the statistical properties of the solution and key quantities derived from it? To answer this question, we present a projector-based approach to sketched linear regression that is exact and that requires minimal assumptions on the sketching matrix. Therefore, downstream analyses hold exactly and generally for all sketching schemes. Additionally, a projector-based approach enables derivation of key quantities from classic linear regression that account for the combined model- and algorithm-induced uncertainties. We demonstrate the usefulness of a projector-based approach in quantifying and enabling insight on excess uncertainties and bias-variance decompositions for sketched linear regression. Finally, we demonstrate how the insights from our projector-based analyses can be used to produce practical sketching diagnostics to aid the design of judicious sketching schemes.