Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

TL;DR

This paper introduces Neural Parameter Search (NPS-Pruning), a novel method for efficiently slimming fine-tuned models by searching neural parameters within low-rank subspaces, improving transfer, fusion, and compression.

Contribution

The paper proposes NPS-Pruning, a new pruning strategy that leverages task vector differences and low-rank subspace search to enhance fine-tuned model compression and transfer capabilities.

Findings

Effective model slimming across vision, NLP, and multi-modal tasks.

Improved knowledge transfer and model merging performance.

Significant reduction in storage costs with minimal performance loss.

Abstract

Foundation models and their checkpoints have significantly advanced deep learning, boosting performance across various applications. However, fine-tuned models often struggle outside their specific domains and exhibit considerable redundancy. Recent studies suggest that combining a pruned fine-tuned model with the original pre-trained model can mitigate forgetting, reduce interference when merging model parameters across tasks, and improve compression efficiency. In this context, developing an effective pruning strategy for fine-tuned models is crucial. Leveraging the advantages of the task vector mechanism, we preprocess fine-tuned models by calculating the differences between them and the original model. Recognizing that different task vector subspaces contribute variably to model performance, we introduce a novel method called Neural Parameter Search (NPS-Pruning) for slimming down fine-tuned models. This method enhances pruning efficiency by searching through neural parameters of task vectors within low-rank subspaces. Our method has three key applications: enhancing knowledge transfer through pairwise model interpolation, facilitating effective knowledge fusion via model merging, and enabling the deployment of compressed models that retain near-original performance while significantly reducing storage costs. Extensive experiments across vision, NLP, and multi-modal benchmarks demonstrate the effectiveness and robustness of our approach, resulting in substantial performance gains. The code is publicly available at: https://github.com/duguodong7/NPS-Pruning.