Fast nonlinear embeddings via structured matrices

TL;DR

This paper introduces a new framework for fast, structured matrix-based nonlinear embeddings in machine learning, improving computational efficiency and reducing space complexity while maintaining high-quality randomized function approximations.

Contribution

The paper presents a general structured matrix approach that extends existing methods like the Fast Johnson-Lindenstrauss Transform to nonlinear embeddings, with theoretical guarantees.

Findings

Significant reduction in computation time for randomized embeddings.

The structured approach maintains high approximation quality.

Framework generalizes several known structured random matrices.

Abstract

We present a new paradigm for speeding up randomized computations of several frequently used functions in machine learning. In particular, our paradigm can be applied for improving computations of kernels based on random embeddings. Above that, the presented framework covers multivariate randomized functions. As a byproduct, we propose an algorithmic approach that also leads to a significant reduction of space complexity. Our method is based on careful recycling of Gaussian vectors into structured matrices that share properties of fully random matrices. The quality of the proposed structured approach follows from combinatorial properties of the graphs encoding correlations between rows of these structured matrices. Our framework covers as special cases already known structured approaches such as the Fast Johnson-Lindenstrauss Transform, but is much more general since it can be applied also to highly nonlinear embeddings. We provide strong concentration results showing the quality of the presented paradigm.