ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations

TL;DR

ProcrustesGPT introduces a method to compress large language models by applying orthogonal transformations to structured matrices, enhancing their compressibility without extensive fine-tuning, thus reducing resource requirements.

Contribution

The paper proposes leveraging orthogonal transformations to improve the compressibility of LLM weights within structured matrix classes, enabling more efficient model compression.

Findings

Orthogonal transformations significantly improve structured matrix representation accuracy.

The approach reduces model size while maintaining performance.

Applicable to various structured matrix types supporting efficient projections.

Abstract

Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by structured matrices without any fine-tuning. To overcome this issue, we utilize the fact that LLM output is invariant under certain orthogonal transformations of weight matrices. This insight can be leveraged to identify transformations that significantly improve the compressibility of weights within structured classes. The proposed approach is applicable to various types of structured matrices that support efficient projection operations. Code is available at https://github.com/GrishKate/ProcrustesGPT