Lossless Model Compression via Joint Low-Rank Factorization Optimization

TL;DR

This paper introduces a joint low-rank factorization optimization method that achieves lossless model compression, surpassing original performance without fine-tuning, applicable across vision and language models.

Contribution

It presents a novel joint optimization strategy for low-rank weight factorization that guarantees lossless compression and improved performance, unlike previous separate optimization approaches.

Findings

Achieves 70% compression on ResNext50 with better performance than original

Develops algorithms that do not require fine-tuning for lossless compression

Demonstrates robustness across various vision and language tasks

Abstract

Low-rank factorization is a popular model compression technique that minimizes the error $\delta$ between approximated and original weight matrices. Despite achieving performances close to the original models when $\delta$ is optimized, a performance discrepancy remains due to the separate optimization processes for low-rank factorization and model performance, resulting in unavoidable losses. We address this issue by introducing a novel joint optimization strategy for lossless low-rank weight factorization, which, for the first time, enhances the model's performance beyond the original. Our approach begins with a theoretical analysis of the relationship between low-rank factorization and model optimization objectives, establishing a precise perturbation range for matrix factorization errors on model performance. This challenge is then reformulated as a numerical rank deficiency problem with inequality constraints and develop a joint objective that simultaneously addresses factorization error and model performance. Based on the above analysis, we propose two optimization algorithms: \textbf{a lossless optimization algorithm} that maximizes model accuracy while ensuring compression, and \textbf{a compact optimization algorithm} that minimizes model size while preserving performance. These algorithms do not require fine-tuning and can directly compress numerous deep models to achieve lossless results. Our methods demonstrate robust efficacy across various vision and language tasks. For example, the compressed model reduced by 70\% on ResNext50 outperforms the original. Our code will be made public.