When big data actually are low-rank, or entrywise approximation of certain function-generated matrices

TL;DR

This paper clarifies misconceptions about low-rank approximations of function-generated matrices, showing that certain classes can be approximated accurately with low rank independent of data dimension, with implications for big data and neural networks.

Contribution

The paper provides a theoretical explanation for when function-generated matrices can be approximated with low rank independent of the ambient dimension, extending to tensor-train formats.

Findings

Low-rank approximation is possible for specific function classes.

Rank depends logarithmically on matrix size, not dimension.

Results apply to big data matrices and neural network attention mechanisms.

Abstract

The article concerns low-rank approximation of matrices generated by sampling a smooth function of two $m$-dimensional variables. We identify several misconceptions surrounding a claim that, for a specific class of analytic functions, such $n \times n$ matrices admit accurate entrywise approximation of rank that is independent of $m$ and grows as $\log(n)$ -- colloquially known as ''big-data matrices are approximately low-rank''. We provide a theoretical explanation of the numerical results presented in support of this claim, describing three narrower classes of functions for which function-generated matrices can be approximated within an entrywise error of order $\varepsilon$ with rank $\mathcal{O}(\log(n) \varepsilon^{-2} \log(\varepsilon^{-1}))$ that is independent of the dimension $m$: (i) functions of the inner product of the two variables, (ii) functions of the Euclidean distance between the variables, and (iii) shift-invariant positive-definite kernels. We extend our argument to tensor-train approximation of tensors generated with functions of the ''higher-order inner product'' of their multiple variables. We discuss our results in the context of low-rank approximation of (a) growing datasets and (b) attention in transformer neural networks.