Learning Sparsity and Randomness for Data-driven Low Rank Approximation

TL;DR

This paper introduces Learning Sparsity and Learning Randomness methods to enhance data-driven low rank approximation by optimizing sketch matrix patterns, improving accuracy and robustness.

Contribution

It proposes novel techniques to learn sparsity patterns and incorporate randomness into sketch matrices, addressing out-of-distribution performance issues.

Findings

Improved test error in low rank approximation

Enhanced out-of-distribution robustness

Applicable to existing learning-based algorithms

Abstract

Learning-based low rank approximation algorithms can significantly improve the performance of randomized low rank approximation with sketch matrix. With the learned value and fixed non-zero positions for sketch matrices from learning-based algorithms, these matrices can reduce the test error of low rank approximation significantly. However, there is still no good method to learn non-zero positions as well as overcome the out-of-distribution performance loss. In this work, we introduce two new methods Learning Sparsity and Learning Randomness which try to learn a better sparsity patterns and add randomness to the value of sketch matrix. These two methods can be applied with any learning-based algorithms which use sketch matrix directly. Our experiments show that these two methods can improve the performance of previous learning-based algorithm for both test error and out-of-distribution test error without adding too much complexity.