Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinement

TL;DR

This paper introduces Singular Value Scaling (SVS), a novel technique for refining pruned weights in generative models, enhancing fine-tuning efficiency and performance across both GANs and Diffusion models without extra training.

Contribution

The paper presents a versatile SVS method that improves weight initialization and fine-tuning in pruned generative models, applicable to multiple architectures.

Findings

SVS improves compression performance on StyleGAN2, StyleGAN3, and DDPM.

SVS speeds up fine-tuning and enhances model quality.

The method works without additional training costs.

Abstract

While pruning methods effectively maintain model performance without extra training costs, they often focus solely on preserving crucial connections, overlooking the impact of pruned weights on subsequent fine-tuning or distillation, leading to inefficiencies. Moreover, most compression techniques for generative models have been developed primarily for GANs, tailored to specific architectures like StyleGAN, and research into compressing Diffusion models has just begun. Even more, these methods are often applicable only to GANs or Diffusion models, highlighting the need for approaches that work across both model types. In this paper, we introduce Singular Value Scaling (SVS), a versatile technique for refining pruned weights, applicable to both model types. Our analysis reveals that pruned weights often exhibit dominant singular vectors, hindering fine-tuning efficiency and leading to suboptimal performance compared to random initialization. Our method enhances weight initialization by minimizing the disparities between singular values of pruned weights, thereby improving the fine-tuning process. This approach not only guides the compressed model toward superior solutions but also significantly speeds up fine-tuning. Extensive experiments on StyleGAN2, StyleGAN3 and DDPM demonstrate that SVS improves compression performance across model types without additional training costs. Our code is available at: https://github.com/LAIT-CVLab/Singular-Value-Scaling.