Fundamental Bias in Inverting Random Sampling Matrices with Application to Sub-sampled Newton

TL;DR

This paper investigates the bias introduced when inverting random sampling matrices in machine learning and numerical linear algebra, proposing correction methods for various sampling and projection techniques to improve sub-sampled Newton methods.

Contribution

It introduces bias correction techniques for inverse of random sampling matrices, including leverage-based and structured projections, enhancing convergence analysis of sub-sampled Newton methods.

Findings

Bias correction methods for sampling matrices are effective.

Corrected inverses improve convergence rates in sub-sampled Newton.

The approach applies to various sampling and projection schemes.

Abstract

A substantial body of work in machine learning (ML) and randomized numerical linear algebra (RandNLA) has exploited various sorts of random sketching methodologies, including random sampling and random projection, with much of the analysis using Johnson--Lindenstrauss and subspace embedding techniques. Recent studies have identified the issue of inversion bias -- the phenomenon that inverses of random sketches are not unbiased, despite the unbiasedness of the sketches themselves. This bias presents challenges for the use of random sketches in various ML pipelines, such as fast stochastic optimization, scalable statistical estimators, and distributed optimization. In the context of random projection, the inversion bias can be easily corrected for dense Gaussian projections (which are, however, too expensive for many applications). Recent work has shown how the inversion bias can be corrected for sparse sub-gaussian projections. In this paper, we show how the inversion bias can be corrected for random sampling methods, both uniform and non-uniform leverage-based, as well as for structured random projections, including those based on the Hadamard transform. Using these results, we establish problem-independent local convergence rates for sub-sampled Newton methods.