The Unreasonable Effectiveness of Structured Random Orthogonal Embeddings

TL;DR

This paper explores structured random orthogonal embeddings that improve accuracy and speed in machine learning tasks like dimensionality reduction and kernel approximation, introducing complex matrices and providing theoretical insights and empirical validation.

Contribution

It introduces new structured random orthogonal matrices, including complex entries, that enhance performance in embeddings for machine learning applications.

Findings

Improved accuracy and speed over previous methods

Complex matrices yield significant accuracy gains

Empirical results confirm wider applicability

Abstract

We examine a class of embeddings based on structured random matrices with orthogonal rows which can be applied in many machine learning applications including dimensionality reduction and kernel approximation. For both the Johnson-Lindenstrauss transform and the angular kernel, we show that we can select matrices yielding guaranteed improved performance in accuracy and/or speed compared to earlier methods. We introduce matrices with complex entries which give significant further accuracy improvement. We provide geometric and Markov chain-based perspectives to help understand the benefits, and empirical results which suggest that the approach is helpful in a wider range of applications.