Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization

TL;DR

This paper introduces a Bayesian optimization-based low-rank compression method for large language models, effectively balancing efficiency and performance by accurately estimating feature distributions and allocating low-rank dimensions.

Contribution

It proposes a novel low-rank compression technique tailored for LLMs, utilizing Bayesian optimization for optimal dimension allocation based on empirical feature distribution analysis.

Findings

Outperforms existing compression methods in maintaining model performance

Effectively estimates feature distributions using pooled covariance matrices

Achieves better compression-performance trade-off on LLaMA-2 models

Abstract

In recent years, large language models (LLMs) have driven advances in natural language processing. Still, their growing scale has increased the computational burden, necessitating a balance between efficiency and performance. Low-rank compression, a promising technique, reduces non-essential parameters by decomposing weight matrices into products of two low-rank matrices. Yet, its application in LLMs has not been extensively studied. The key to low-rank compression lies in low-rank factorization and low-rank dimensions allocation. To address the challenges of low-rank compression in LLMs, we conduct empirical research on the low-rank characteristics of large models. We propose a low-rank compression method suitable for LLMs. This approach involves precise estimation of feature distributions through pooled covariance matrices and a Bayesian optimization strategy for allocating low-rank dimensions. Experiments on the LLaMA-2 models demonstrate that our method outperforms existing strong structured pruning and low-rank compression techniques in maintaining model performance at the same compression ratio.