Language model compression with weighted low-rank factorization

TL;DR

This paper introduces Fisher-Weighted SVD (FWSVD), a novel matrix factorization method for language model compression that aligns the approximation process with task importance, resulting in better performance retention at higher compression rates.

Contribution

The paper proposes FWSVD, which incorporates Fisher information into SVD to prioritize important parameters, improving task accuracy preservation during model compression.

Findings

FWSVD maintains higher task accuracy compared to traditional SVD.

The method achieves 9-30% parameter reduction with minimal accuracy loss.

FWSVD outperforms other compact strategies without requiring extensive pre-training.

Abstract

Factorizing a large matrix into small matrices is a popular strategy for model compression. Singular value decomposition (SVD) plays a vital role in this compression strategy, approximating a learned matrix with fewer parameters. However, SVD minimizes the squared error toward reconstructing the original matrix without gauging the importance of the parameters, potentially giving a larger reconstruction error for those who affect the task accuracy more. In other words, the optimization objective of SVD is not aligned with the trained model's task accuracy. We analyze this previously unexplored problem, make observations, and address it by introducing Fisher information to weigh the importance of parameters affecting the model prediction. This idea leads to our method: Fisher-Weighted SVD (FWSVD). Although the factorized matrices from our approach do not result in smaller reconstruction errors, we find that our resulting task accuracy is much closer to the original model's performance. We perform analysis with the transformer-based language models, showing our weighted SVD largely alleviates the mismatched optimization objectives and can maintain model performance with a higher compression rate. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies requiring expensive model pre-training. Moreover, the evaluation of compressing an already compact model shows our method can further reduce 9% to 30% parameters with an insignificant impact on task accuracy.