Sparser, Faster, Lighter Transformer Language Models

TL;DR

This paper introduces a method to make large language models sparser, faster, and lighter by using unstructured sparsity and specialized GPU kernels, significantly improving efficiency with minimal performance loss.

Contribution

The authors develop a new sparse packing format and CUDA kernels that enable efficient sparse computation in LLMs, demonstrating high sparsity levels with negligible impact on performance.

Findings

Over 99% sparsity achieved with minimal performance impact

Significant throughput, energy, and memory benefits demonstrated

Open-source code and kernels released for community adoption

Abstract

Scaling autoregressive large language models (LLMs) has driven unprecedented progress but comes with vast computational costs. In this work, we tackle these costs by leveraging unstructured sparsity within an LLM's feedforward layers, the components accounting for most of the model parameters and execution FLOPs. To achieve this, we introduce a new sparse packing format and a set of CUDA kernels designed to seamlessly integrate with the optimized execution pipelines of modern GPUs, enabling efficient sparse computation during LLM inference and training. To substantiate our gains, we provide a quantitative study of LLM sparsity, demonstrating that simple L1 regularization can induce over 99% sparsity with negligible impact on downstream performance. When paired with our kernels, we show that these sparsity levels translate into substantial throughput, energy efficiency, and memory usage benefits that increase with model scale. We will release all code and kernels under an open-source license to promote adoption and accelerate research toward establishing sparsity as a practical axis for improving the efficiency and scalability of modern foundation models.