Sketching Transformed Matrices with Applications to Natural Language Processing

TL;DR

This paper introduces a space-efficient sketching method for computing products involving large, transformed matrices, enabling scalable matrix decompositions in NLP applications like word embeddings.

Contribution

The paper presents a novel sketching algorithm for transformed matrices that is space-efficient, generalizable to various functions, and applicable to low-rank approximation tasks.

Findings

The sketching method achieves small approximation error.

It is efficient in both space and computational time.

Experimental results validate the theoretical guarantees.

Abstract

Suppose we are given a large matrix $A=(a_{i,j})$ that cannot be stored in memory but is in a disk or is presented in a data stream. However, we need to compute a matrix decomposition of the entry-wisely transformed matrix, $f(A):=(f(a_{i,j}))$ for some function $f$. Is it possible to do it in a space efficient way? Many machine learning applications indeed need to deal with such large transformed matrices, for example word embedding method in NLP needs to work with the pointwise mutual information (PMI) matrix, while the entrywise transformation makes it difficult to apply known linear algebraic tools. Existing approaches for this problem either need to store the whole matrix and perform the entry-wise transformation afterwards, which is space consuming or infeasible, or need to redesign the learning method, which is application specific and requires substantial remodeling. In this paper, we first propose a space-efficient sketching algorithm for computing the product of a given small matrix with the transformed matrix. It works for a general family of transformations with provable small error bounds and thus can be used as a primitive in downstream learning tasks. We then apply this primitive to a concrete application: low-rank approximation. We show that our approach obtains small error and is efficient in both space and time. We complement our theoretical results with experiments on synthetic and real data.