Fast Generalized Matrix Regression with Applications in Machine Learning

TL;DR

This paper introduces a fast generalized matrix regression algorithm using sketching techniques, enabling efficient large-scale matrix approximations with provable error bounds and improved performance over traditional methods.

Contribution

The paper presents a novel fast GMR algorithm that achieves $(1+ ext{epsilon})$ relative error with small sketch sizes, applicable to various matrix approximation tasks.

Findings

Achieves $(1+ ext{epsilon})$ error with $ ext{O}( ext{epsilon}^{-1/2})$ sketch size.

Improves performance in symmetric positive definite matrix approximation.

Outperforms conventional algorithms in empirical tests.

Abstract

Fast matrix algorithms have become the fundamental tools of machine learning in big data era. The generalized matrix regression problem is widely used in the matrix approximation such as CUR decomposition, kernel matrix approximation, and stream singular value decomposition (SVD), etc. In this paper, we propose a fast generalized matrix regression algorithm (Fast GMR) which utilizes sketching technique to solve the GMR problem efficiently. Given error parameter $0<\epsilon<1$, the Fast GMR algorithm can achieve a $(1+\epsilon)$ relative error with the sketching sizes being of order $\cO(\epsilon^{-1/2})$ for a large group of GMR problems. We apply the Fast GMR algorithm to the symmetric positive definite matrix approximation and single pass singular value decomposition and they achieve a better performance than conventional algorithms. Our empirical study also validates the effectiveness and efficiency of our proposed algorithms.