Revisiting Co-Occurring Directions: Sharper Analysis and Efficient Algorithm for Sparse Matrices

TL;DR

This paper improves the theoretical understanding and efficiency of the co-occurring directions algorithm for streaming approximate matrix multiplication, especially for sparse matrices, with tighter error bounds and practical benefits.

Contribution

It provides a tighter error analysis for COD, proves its space optimality, and introduces a variant optimized for sparse matrices with theoretical guarantees.

Findings

Tighter error bounds considering low-rank structure and matrix correlation.

COD is proven to be space optimal under the new bounds.

The sparse matrix variant outperforms baseline methods in efficiency.

Abstract

We study the streaming model for approximate matrix multiplication (AMM). We are interested in the scenario that the algorithm can only take one pass over the data with limited memory. The state-of-the-art deterministic sketching algorithm for streaming AMM is the co-occurring directions (COD), which has much smaller approximation errors than randomized algorithms and outperforms other deterministic sketching methods empirically. In this paper, we provide a tighter error bound for COD whose leading term considers the potential approximate low-rank structure and the correlation of input matrices. We prove COD is space optimal with respect to our improved error bound. We also propose a variant of COD for sparse matrices with theoretical guarantees. The experiments on real-world sparse datasets show that the proposed algorithm is more efficient than baseline methods.