Higher-order Count Sketch: Dimensionality Reduction That Retains Efficient Tensor Operations

TL;DR

Higher-order Count Sketch (HCS) extends traditional count sketch by using multiple hash functions and tensor reshaping, enabling efficient tensor operations and significant data compression while maintaining high accuracy in neural network applications.

Contribution

HCS introduces a novel tensor-based sketching method that exploits multi-dimensional data structures for improved efficiency and accuracy in large-scale tensor computations.

Findings

HCS achieves exponential memory savings compared to count sketch.

HCS enables efficient approximation of tensor operations directly on sketched data.

HCS maintains high accuracy in tensorized neural networks with substantial compression.

Abstract

Sketching is a randomized dimensionality-reduction method that aims to preserve relevant information in large-scale datasets. Count sketch is a simple popular sketch which uses a randomized hash function to achieve compression. In this paper, we propose a novel extension known as Higher-order Count Sketch (HCS). While count sketch uses a single hash function, HCS uses multiple (smaller) hash functions for sketching. HCS reshapes the input (vector) data into a higher-order tensor and employs a tensor product of the random hash functions to compute the sketch. This results in an exponential saving (with respect to the order of the tensor) in the memory requirements of the hash functions, under certain conditions on the input data. Furthermore, when the input data itself has an underlying structure in the form of various tensor representations such as the Tucker decomposition, we obtain significant advantages. We derive efficient (approximate) computation of various tensor operations such as tensor products and tensor contractions directly on the sketched data. Thus, HCS is the first sketch to fully exploit the multi-dimensional nature of higher-order tensors. We apply HCS to tensorized neural networks where we replace fully connected layers with sketched tensor operations. We achieve nearly state of the art accuracy with significant compression on the image classification benchmark.