EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training

TL;DR

EDGC introduces an entropy-driven dynamic gradient compression method that adaptively adjusts compression rates during large language model training, significantly reducing communication overhead while maintaining model accuracy.

Contribution

The paper presents a novel entropy-based dynamic gradient compression framework that adjusts compression rates during training based on gradient entropy trends, improving efficiency and performance.

Findings

Reduces communication latency by up to 46.45%.

Decreases training time by up to 16.13%.

Maintains model accuracy with adaptive compression.

Abstract

Training large language models (LLMs) poses significant challenges regarding computational resources and memory capacity. Although distributed training techniques help mitigate these issues, they still suffer from considerable communication overhead. Existing approaches primarily rely on static gradient compression to enhance communication efficiency; however, these methods neglect the dynamic nature of evolving gradients during training, leading to performance degradation. Accelerating LLM training via compression without sacrificing performance remains a challenge. In this paper, we propose an entropy-driven dynamic gradient compression framework called EDGC. The core concept is to adjust the compression rate during LLM training based on the evolving trends of gradient entropy, taking into account both compression efficiency and error. EDGC consists of three key components.First, it employs a down-sampling method to efficiently estimate gradient entropy, reducing computation overhead. Second, it establishes a theoretical model linking compression rate with gradient entropy, enabling more informed compression decisions. Lastly, a window-based adjustment mechanism dynamically adapts the compression rate across pipeline stages, improving communication efficiency and maintaining model performance. We implemented EDGC on a 32-NVIDIA-V100 cluster and a 64-NVIDIA-H100 cluster to train GPT2-2.5B and GPT2-12.1B, respectively. The results show that EDGC significantly reduces communication latency and training time by up to 46.45% and 16.13% while preserving LLM accuracy.