Cardinality Sparsity: Applications in Matrix-Matrix Multiplications and Machine Learning

TL;DR

This paper introduces cardinality sparsity, a new form of sparsity based on the number of unique values in data, which enhances deep learning, tensor regression, and significantly accelerates matrix-matrix multiplication while reducing memory usage.

Contribution

It defines and explores cardinality sparsity, demonstrating its benefits in statistical modeling, computational efficiency, and memory reduction, especially in matrix operations.

Findings

Cardinality sparsity improves deep learning and tensor regression performance.

Algorithms leveraging cardinality sparsity outperform existing matrix multiplication methods.

Memory usage is significantly reduced by executing matrix multiplication in the compressed domain.

Abstract

High-dimensional data has become ubiquitous across the sciences but presents computational and statistical challenges. A common approach to addressing these challenges is through sparsity. In this paper, we introduce a new concept of sparsity, called cardinality sparsity. Broadly speaking, we define a tensor as sparse if it contains only a small number of unique values. We demonstrate that cardinality sparsity can improve deep learning and tensor regression both statistically and computationally. Along the way, we generalize recent statistical theories in these fields. Most importantly, we show that cardinality sparsity has a strikingly powerful application beyond high-dimensional data analysis: it can significantly speed up matrix-matrix multiplications. For instance, we demonstrate that cardinality sparsity leads to algorithms for binary-matrix multiplication that outperform state-of-the-art algorithms by a substantial margin. Additionally, another crucial aspect of this sparsity is minimizing memory usage. By executing matrix multiplication in the compressed domain, we can significantly lower the amount of memory needed to store the input data.