Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks

TL;DR

This paper introduces Meta-Vote Pruning (MVP), a method that leverages pruned models of similar tasks to efficiently create effective models for new downstream tasks, reducing pruning iterations and improving performance.

Contribution

The paper proposes a novel meta-pruning approach that reuses pruned models from similar tasks to accelerate and improve pruning for new tasks in pre-trained models.

Findings

MVP significantly reduces pruning iterations needed for new tasks.

Pruned models for similar tasks have overlapping structures that can be exploited.

MVP achieves better accuracy and efficiency compared to traditional pruning methods.

Abstract

As a few large-scale pre-trained models become the major choices of various applications, new challenges arise for model pruning, e.g., can we avoid pruning the same model from scratch for every downstream task? How to reuse the pruning results of previous tasks to accelerate the pruning for a new task? To address these challenges, we create a small model for a new task from the pruned models of similar tasks. We show that a few fine-tuning steps on this model suffice to produce a promising pruned-model for the new task. We study this ''meta-pruning'' from nearest tasks on two major classes of pre-trained models, convolutional neural network (CNN) and vision transformer (ViT), under a limited budget of pruning iterations. Our study begins by investigating the overlap of pruned models for similar tasks and how the overlap changes over different layers and blocks. Inspired by these discoveries, we develop a simple but effective ''Meta-Vote Pruning (MVP)'' method that significantly reduces the pruning iterations for a new task by initializing a sub-network from the pruned models of its nearest tasks. In experiments, we demonstrate MVP's advantages in accuracy, efficiency, and generalization through extensive empirical studies and comparisons with popular pruning methods over several datasets.