Sparse sketches with small inversion bias
Micha{\l} Derezi\'nski, Zhenyu Liao, Edgar Dobriban, Michael W., Mahoney

TL;DR
This paper introduces a new framework for analyzing inversion bias in matrix sketching, proposes a novel LESS embedding technique, and demonstrates its effectiveness in reducing bias with efficient computation.
Contribution
The paper develops an $( ext{epsilon}, ext{delta})$-unbiased estimator framework, introduces LESS embeddings, and provides theoretical guarantees for reduced inversion bias in matrix sketching.
Findings
The estimator $(rac m{m-d} ilde A^ op ilde A)^{-1}$ has $O(1/\sqrt d)$ inversion bias for $m=O(d)$.
LESS embeddings achieve $ ext{epsilon}$ inversion bias with sketch size $O(d ext{log} d + rac{ ext{sqrt} d}{ ext{epsilon}})$.
The analysis extends classical inequalities and shows limitations of leverage score sampling for small inversion bias.
Abstract
For a tall matrix and a random sketching matrix , the sketched estimate of the inverse covariance matrix is typically biased: , where . This phenomenon, which we call inversion bias, arises, e.g., in statistics and distributed optimization, when averaging multiple independently constructed estimates of quantities that depend on the inverse covariance. We develop a framework for analyzing inversion bias, based on our proposed concept of an -unbiased estimator for random matrices. We show that when the sketching matrix is dense and has i.i.d. sub-gaussian entries, then after simple rescaling, the estimator is -unbiased for with a sketch of size . This…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSparse and Compressive Sensing Techniques · Random Matrices and Applications · Stochastic Gradient Optimization Techniques
