On Using Toeplitz and Circulant Matrices for Johnson-Lindenstrauss Transforms

TL;DR

This paper proves that using Toeplitz matrices for Johnson-Lindenstrauss embeddings requires at least O(ε^{-2} log^2 N) dimensions, confirming the current upper bounds and indicating no faster embedding dimension is achievable with this method.

Contribution

It establishes a lower bound matching existing upper bounds for Toeplitz-based Johnson-Lindenstrauss transforms, showing the current analysis is tight.

Findings

Toeplitz matrices require Ω(ε^{-2} log^2 N) dimensions for JL embeddings.

Current best analysis showing O(ε^{-2} log^2 N) suffices is tight.

No faster dimension reduction using Toeplitz matrices is possible beyond this bound.

Abstract

The Johnson-Lindenstrauss lemma is one of the corner stone results in dimensionality reduction. It says that given $N$, for any set of $N$ vectors $X \subset \mathbb{R}^n$, there exists a mapping $f : X \to \mathbb{R}^m$ such that $f(X)$ preserves all pairwise distances between vectors in $X$ to within $(1 \pm \varepsilon)$ if $m = O(\varepsilon^{-2} \lg N)$. Much effort has gone into developing fast embedding algorithms, with the Fast Johnson-Lindenstrauss transform of Ailon and Chazelle being one of the most well-known techniques. The current fastest algorithm that yields the optimal $m = O(\varepsilon^{-2}\lg N)$ dimensions has an embedding time of $O(n \lg n + \varepsilon^{-2} \lg^3 N)$. An exciting approach towards improving this, due to Hinrichs and Vyb\'iral, is to use a random $m \times n$ Toeplitz matrix for the embedding. Using Fast Fourier Transform, the embedding of a vector can then be computed in $O(n \lg m)$ time. The big question is of course whether $m = O(\varepsilon^{-2} \lg N)$ dimensions suffice for this technique. If so, this would end a decades long quest to obtain faster and faster Johnson-Lindenstrauss transforms. The current best analysis of the embedding of Hinrichs and Vyb\'iral shows that $m = O(\varepsilon^{-2}\lg^2 N)$ dimensions suffices. The main result of this paper, is a proof that this analysis unfortunately cannot be tightened any further, i.e., there exists a set of $N$ vectors requiring $m = \Omega(\varepsilon^{-2} \lg^2 N)$ for the Toeplitz approach to work.