SWSC: Shared Weight for Similar Channel in LLM

TL;DR

SWSC is a novel LLM compression technique that clusters similar weights to reduce parameters and uses SVD-based compensation to maintain performance, enabling efficient deployment of large models.

Contribution

The paper introduces SWSC, a new compression method combining clustering and SVD-based correction to effectively reduce LLM size while preserving accuracy.

Findings

Significant parameter reduction achieved with minimal performance loss.

Effective in low-precision settings for LLMs.

Outperforms existing compression methods in experiments.

Abstract

Large language models (LLMs) have spurred development in multiple industries. However, the growing number of their parameters brings substantial storage and computing burdens, making it essential to explore model compression techniques for parameter reduction and easier deployment. We propose SWSC, an LLM compression method based on the concept of Shared Weight for Similar Channel. It uses the K-Means clustering algorithm to cluster model weights channel-by-channel, generating clusters with highly similar vectors within each. A representative vector from each cluster is selected to approximately replace all vectors in the cluster, significantly reducing the number of model weight parameters. However, approximate restoration will inevitably cause damage to the performance of the model. To tackle this issue, we perform singular value decomposition on the weight error values before and after compression and retain the larger singular values and their corresponding singular vectors to compensate for the accuracy. The experimental results show that our method can effectively ensure the performance of the compressed LLM even under low-precision conditions.