Training-Free Activation Sparsity in Large Language Models

TL;DR

TEAL introduces a simple, training-free magnitude-based activation sparsity method for large language models, achieving significant speedups and efficiency improvements with minimal performance loss across various model sizes.

Contribution

It presents TEAL, a novel training-free approach to induce activation sparsity in large language models, enabling faster inference without extensive retraining.

Findings

Achieves 40-50% model-wide sparsity with minimal performance loss.

Provides up to 1.8x decoding speed-up at 50% sparsity.

Compatible with weight quantization for additional efficiency.

Abstract

Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for matrix multiplications during the forward pass. However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. This paper describes TEAL, a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53$\times$ and 1.8$\times$ at 40% and 50% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.