ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity

TL;DR

ELAS introduces a novel low-rank training framework for large language models that leverages 2:4 activation sparsity to reduce memory usage and accelerate training without significant performance loss.

Contribution

ELAS applies 2:4 structured sparsity to activations in low-rank LLMs, enabling efficient pre-training with reduced memory and computational overhead.

Findings

ELAS maintains model performance with minimal degradation.

ELAS accelerates training and inference processes.

ELAS significantly reduces activation memory overhead.

Abstract

Large Language Models (LLMs) have achieved remarkable capabilities, but their immense computational demands during training remain a critical bottleneck for widespread adoption. Low-rank training has received attention in recent years due to its ability to significantly reduce training memory usage. Meanwhile, applying 2:4 structured sparsity to weights and activations to leverage NVIDIA GPU support for 2:4 structured sparse format has become a promising direction. However, existing low-rank methods often leave activation matrices in full-rank, which dominates memory consumption and limits throughput during large-batch training. Furthermore, directly applying sparsity to weights often leads to non-negligible performance degradation. To achieve efficient pre-training of LLMs, this paper proposes ELAS: Efficient pre-training of Low-rank LLMs via 2:4 Activation Sparsity, a novel framework for low-rank models via 2:4 activation sparsity. ELAS applies squared ReLU activation functions to the feed-forward networks in low-rank models and implements 2:4 structured sparsity on the activations after the squared ReLU operation. We evaluated ELAS through pre-training experiments on LLaMA models ranging from 60M to 1B parameters. The results demonstrate that ELAS maintains performance with minimal degradation after applying 2:4 activation sparsity, while achieving training and inference acceleration. Moreover, ELAS reduces activation memory overhead, particularly with large batch sizes. Code is available at ELAS Repo.