Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models

TL;DR

This paper introduces LM-GC, a novel method using large language models as zero-shot gradient priors for lossless compression, significantly improving compression efficiency and demonstrating potential for neural network gradient modeling.

Contribution

The paper presents LM-GC, a new approach that leverages LLMs with arithmetic coding to enhance gradient compression, a novel application of language models in this domain.

Findings

Token efficiency increased by up to 38 times.

Achieved 10% to 17.2% better compression rates than existing methods.

Demonstrated compatibility with lossy compression techniques.

Abstract

Despite the widespread use of statistical prior models in various fields, such models for neural network gradients have long been overlooked. The inherent challenge stems from their high-dimensional structures and complex interdependencies, which complicate effective modeling. In this work, we demonstrate the potential of large language models (LLMs) to act as gradient priors in a zero-shot setting. We examine the property by considering lossless gradient compression -- a critical application in distributed learning -- that depends heavily on precise probability modeling. To achieve this, we introduce LM-GC, a novel method that integrates LLMs with arithmetic coding. Our technique converts plain gradients into text-like formats, enhancing token efficiency by up to 38 times compared to their plain representations. We ensure that this data conversion maintains a close alignment with the structure of plain gradients and the symbols commonly recognized by LLMs. Our experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods, improving compression rates by 10% up to 17.2% across various datasets and architectures. Additionally, our approach shows promising compatibility with lossy compression techniques such as quantization and sparsification. These findings highlight the significant potential of LLMs as a model for effectively handling gradients. Code is available at https://github.com/hui-po-wang/LM-GC.