PELA: Learning Parameter-Efficient Models with Low-Rank Approximation

TL;DR

This paper introduces PELA, a parameter-efficient method that compresses large pre-trained models using low-rank approximation and specialized modules, enabling effective downstream fine-tuning with reduced resources.

Contribution

The paper proposes a novel low-rank approximation approach with feature distillation and regularization modules, updating only the compressed model for efficiency.

Findings

Reduces parameter size by 1/3 to 2/3 with minimal performance loss

Maintains comparable results across multiple vision and vision-language models

Achieves efficiency in parameters and computation time

Abstract

Applying a pre-trained large model to downstream tasks is prohibitive under resource-constrained conditions. Recent dominant approaches for addressing efficiency issues involve adding a few learnable parameters to the fixed backbone model. This strategy, however, leads to more challenges in loading large models for downstream fine-tuning with limited resources. In this paper, we propose a novel method for increasing the parameter efficiency of pre-trained models by introducing an intermediate pre-training stage. To this end, we first employ low-rank approximation to compress the original large model and then devise a feature distillation module and a weight perturbation regularization module. These modules are specifically designed to enhance the low-rank model. In particular, we update only the low-rank model while freezing the backbone parameters during pre-training. This allows for direct and efficient utilization of the low-rank model for downstream fine-tuning tasks. The proposed method achieves both efficiencies in terms of required parameters and computation time while maintaining comparable results with minimal modifications to the backbone architecture. Specifically, when applied to three vision-only and one vision-language Transformer models, our approach often demonstrates a merely $\sim$0.6 point decrease in performance while reducing the original parameter size by 1/3 to 2/3.