A High Performance GPU CountSketch Implementation and Its Application to Multisketching and Least Squares Problems

TL;DR

This paper presents a high-performance GPU implementation of CountSketch, enabling efficient multisketching and improved least squares solutions with significant speedups and numerical stability benefits.

Contribution

It introduces an optimized GPU-based CountSketch implementation and demonstrates its effectiveness in multisketching and least squares problems, outperforming traditional methods.

Findings

GPU CountSketch implementation achieves high efficiency

Multisketching with GPU CountSketch accelerates dimension reduction

Multisketched least squares solver is up to 77% faster with better stability

Abstract

Random sketching is a dimensionality reduction technique that approximately preserves norms and singular values up to some $O(1)$ distortion factor with high probability. The most popular sketches in literature are the Gaussian sketch and the subsampled randomized Hadamard transform, while the CountSketch has lower complexity. Combining two sketches, known as multisketching, offers an inexpensive means of quickly reducing the dimension of a matrix by combining a CountSketch and Gaussian sketch. However, there has been little investigation into high performance CountSketch implementations. In this work, we develop an efficient GPU implementation of the CountSketch, and demonstrate the performance of multisketching using this technique. We also demonstrate the potential for using this implementation within a multisketched least squares solver that is up to $77\%$ faster than the normal equations with significantly better numerical stability, at the cost of an $O(1)$ multiplicative factor introduced into the relative residual norm.