Distributed Least Squares in Small Space via Sketching and Bias Reduction

TL;DR

This paper introduces a bias-focused sketching method for distributed least squares regression, achieving near-unbiased estimators with optimal space and communication efficiency, overcoming fundamental limitations of traditional matrix sketching.

Contribution

It proposes a novel sparse sketching technique that minimizes estimator bias, enabling efficient distributed least squares with improved accuracy and communication costs.

Findings

Achieves nearly-unbiased least squares estimation with two data passes.

Develops a sparse sketching method with optimal space and matrix multiplication time.

Provides new bias analysis and higher-moment inequalities for sketched least squares.

Abstract

Matrix sketching is a powerful tool for reducing the size of large data matrices. Yet there are fundamental limitations to this size reduction when we want to recover an accurate estimator for a task such as least square regression. We show that these limitations can be circumvented in the distributed setting by designing sketching methods that minimize the bias of the estimator, rather than its error. In particular, we give a sparse sketching method running in optimal space and current matrix multiplication time, which recovers a nearly-unbiased least squares estimator using two passes over the data. This leads to new communication-efficient distributed averaging algorithms for least squares and related tasks, which directly improve on several prior approaches. Our key novelty is a new bias analysis for sketched least squares, giving a sharp characterization of its dependence on the sketch sparsity. The techniques include new higher-moment restricted Bai-Silverstein inequalities, which are of independent interest to the non-asymptotic analysis of deterministic equivalents for random matrices that arise from sketching.