Towards More Efficient SPSD Matrix Approximation and CUR Matrix Decomposition

TL;DR

This paper introduces a fast SPSD matrix approximation model that combines the efficiency of Nyström with the accuracy of the prototype model, enabling near-linear time eigenvalue and kernel learning computations.

Contribution

A novel fast SPSD matrix approximation model that achieves high accuracy with near-linear time complexity, improving upon existing methods like Nyström and prototype models.

Findings

Fast model achieves 1+ε relative-error in linear time.

Empirical results show the fast model outperforms traditional methods.

Nyström is a special case of the proposed fast model.

Abstract

Symmetric positive semi-definite (SPSD) matrix approximation methods have been extensively used to speed up large-scale eigenvalue computation and kernel learning methods. The standard sketch based method, which we call the prototype model, produces relatively accurate approximations, but is inefficient on large square matrices. The Nystr\"om method is highly efficient, but can only achieve low accuracy. In this paper we propose a novel model that we call the {\it fast SPSD matrix approximation model}. The fast model is nearly as efficient as the Nystr\"om method and as accurate as the prototype model. We show that the fast model can potentially solve eigenvalue problems and kernel learning problems in linear time with respect to the matrix size $n$ to achieve $1+\epsilon$ relative-error, whereas both the prototype model and the Nystr\"om method cost at least quadratic time to attain comparable error bound. Empirical comparisons among the prototype model, the Nystr\"om method, and our fast model demonstrate the superiority of the fast model. We also contribute new understandings of the Nystr\"om method. The Nystr\"om method is a special instance of our fast model and is approximation to the prototype model. Our technique can be straightforwardly applied to make the CUR matrix decomposition more efficiently computed without much affecting the accuracy.