Toward Efficient SpMV in Sparse LLMs via Block Extraction and Compressed Storage

TL;DR

This paper introduces EC-SpMV, a GPU-optimized method for sparse LLM inference that leverages hierarchical block extraction and a new compressed format to significantly improve speed and reduce storage overhead.

Contribution

It proposes EC-SpMV, a novel GPU-based approach with hierarchical block extraction and EC-CSR format tailored for sparse LLMs, outperforming existing methods.

Findings

Up to 6.44x speedup over state-of-the-art SpMV libraries.

Reduces storage overhead by up to 55.4% compared to CSR.

Effective on real sparse weight matrices from LLaMA and OPT.

Abstract

Sparse Matrix-Vector Multiplication (SpMV) has become a critical performance bottleneck in the local deployment of sparse Large Language Models (LLMs), where inference predominantly operates on workloads during the decoder phase with a batch size of one. Existing SpMV kernels and sparse matrix formats, originally designed for scientific computing, fail to exploit the unique structure patterns inherent in sparse LLMs, resulting in suboptimal performance and excessive storage overhead. This paper presents EC-SpMV, a GPU-optimized SpMV approach for accelerating sparse LLM inference. EC-SpMV introduces (1) a hierarchical block extraction algorithm that captures multiple granularities of block structures within sparse LLMs, and (2) a novel compressed sparse format (EC-CSR) that employs delta indexing to reduce storage overhead and enhance memory access efficiency. Evaluated on real sparse weight matrices from LLaMA and OPT models, EC-SpMV achieves up to 6.44x speedup over state-of-the-art SpMV libraries and reduces storage overhead by up to 55.4% compared to CSR.