Straightforward Layer-wise Pruning for More Efficient Visual Adaptation

TL;DR

This paper introduces SLS, a layer-wise pruning method for PETL models that reduces storage overhead and improves efficiency by evaluating layer importance through clustering metrics, addressing redundancy in transferred models.

Contribution

The paper proposes a simple, storage-efficient layer-wise pruning approach tailored for PETL models, utilizing clustering metrics to inform pruning decisions and improve model efficiency.

Findings

SLS reduces storage overhead compared to traditional pruning methods.

Pruned models achieve a better balance between accuracy and throughput.

SLS enhances speed and accuracy of PETL models in experiments.

Abstract

Parameter-efficient transfer learning (PETL) aims to adapt large pre-trained models using limited parameters. While most PETL approaches update the added parameters and freeze pre-trained weights during training, the minimal impact of task-specific deep layers on cross-domain data poses a challenge as PETL cannot modify them, resulting in redundant model structures. Structural pruning effectively reduces model redundancy; however, common pruning methods often lead to an excessive increase in stored parameters due to varying pruning structures based on pruning rates and data. Recognizing the storage parameter volume issue, we propose a Straightforward layer-wise pruning method, called SLS, for pruning PETL-transferred models. By evaluating parameters from a feature perspective of each layer and utilizing clustering metrics to assess current parameters based on clustering phenomena in low-dimensional space obtained through t-SNE, SLS facilitates informed pruning decisions. Our study reveals that layer-wise pruning, with a focus on storing pruning indices, addresses storage volume concerns. Notably, mainstream Layer-wise pruning methods may not be suitable for assessing layer importance in PETL-transferred models, where the majority of parameters are pre-trained and have limited relevance to downstream datasets. Comparative analysis against state-of-the-art PETL methods demonstrates that the pruned model achieved a notable balance between model throughput and accuracy. Moreover, SLS effectively reduces storage overhead arising from varying pruned structures while enhancing the accuracy and speed of pruned models compared to conventional pruning methods.