Sparse Random Matrices for Dimensionality Reduction

TL;DR

This paper reviews sparse random matrices for dimensionality reduction, compares their theoretical properties and practical performance against dense Gaussian matrices, and discusses their advantages in computational efficiency.

Contribution

It provides an overview of sparse JL constructions, implements them, and empirically evaluates their effectiveness compared to traditional dense matrices.

Findings

Sparse matrices enable faster matrix-vector multiplication.

Empirical results show comparable distance preservation to dense matrices.

Sparse constructions are practical for large-scale applications.

Abstract

The Johnson-Lindenstrauss (JL) theorem states that a set of points in high-dimensional space can be embedded into a lower-dimensional space while approximately preserving pairwise distances with high probability Johnson and Lindenstrauss (1984). The standard JL theorem uses dense random matrices with Gaussian entries. However, for some applications, sparse random matrices are preferred as they allow for faster matrix-vector multiplication. I outline the constructions and proofs introduced by Achlioptas (2003) and the contemporary standard by Kane and Nelson (2014). Further, I implement and empirically compare these sparse constructions with standard Gaussian JL matrices.