Range-Net: A High Precision Streaming SVD for Big Data Applications

TL;DR

Range-Net introduces a deterministic neural approach for streaming SVD that guarantees tail-energy bounds and significantly outperforms existing randomized methods in accuracy, with efficient memory usage suitable for big data applications.

Contribution

The paper presents Range-Net, a novel neural optimization method for streaming SVD that guarantees EYM tail-energy bounds and improves accuracy over randomized SVD schemes.

Findings

Range-Net achieves six orders of magnitude better accuracy than state-of-the-art streaming randomized SVD.

Range-Net's memory requirement depends only on feature dimension and rank, not sample size.

Theoretical guarantees ensure Range-Net's SVD factors satisfy EYM tail-energy lower bounds.

Abstract

In a Big Data setting computing the dominant SVD factors is restrictive due to the main memory requirements. Recently introduced streaming Randomized SVD schemes work under the restrictive assumption that the singular value spectrum of the data has exponential decay. This is seldom true for any practical data. Although these methods are claimed to be applicable to scientific computations due to associated tail-energy error bounds, the approximation errors in the singular vectors and values are high when the aforementioned assumption does not hold. Furthermore from a practical perspective, oversampling can still be memory intensive or worse can exceed the feature dimension of the data. To address these issues, we present Range-Net as an alternative to randomized SVD that satisfies the tail-energy lower bound given by Eckart-Young-Mirsky (EYM) theorem. Range-Net is a deterministic two-stage neural optimization approach with random initialization, where the main memory requirement depends explicitly on the feature dimension and desired rank, independent of the sample dimension. The data samples are read in a streaming setting with the network minimization problem converging to the desired rank-r approximation. Range-Net is fully interpretable where all the network outputs and weights have a specific meaning. We provide theoretical guarantees that Range-Net extracted SVD factors satisfy EYM tail-energy lower bound at machine precision. Our numerical experiments on real data at various scales confirms this bound. A comparison against the state of the art streaming Randomized SVD shows that Range-Net accuracy is better by six orders of magnitude while being memory efficient.