CompAct: Compressed Activations for Memory-Efficient LLM Training

TL;DR

CompAct introduces a method to significantly reduce GPU memory usage during large language model training by compressing activations with random projections, enabling more efficient training and fine-tuning.

Contribution

The paper presents a novel activation compression technique using random projections that reduces peak memory during LLM training, surpassing prior methods.

Findings

Reduces peak memory by 25-30% in pretraining

Achieves 50% memory reduction during fine-tuning

Improves compute-performance tradeoffs over previous techniques

Abstract

We introduce CompAct, a technique that reduces peak memory utilization on GPU by 25-30% for pretraining and 50% for fine-tuning of LLMs. Peak device memory is a major limiting factor in training LLMs, with various recent works aiming to reduce model memory. However most works don't target the largest component of allocated memory during training: the model's compute graph, which is stored for the backward pass. By storing low-rank, compressed activations to be used in the backward pass we greatly reduce the required memory, unlike previous methods which only reduce optimizer overheads or the number of trained parameters. Our compression uses random projection matrices, thus avoiding additional memory overheads. Comparisons with previous techniques for either pretraining or fine-tuning show that CompAct substantially improves existing compute-performance tradeoffs. We expect CompAct's savings to scale even higher for larger models.