Task-specific Compression for Multi-task Language Models using Attribution-based Pruning

TL;DR

This paper introduces a training-free, attribution-based pruning method to compress multi-task language models by removing unnecessary neurons for specific tasks, preserving performance and reducing resource usage.

Contribution

It presents a novel, training-free pruning approach that selectively removes neurons based on attribution scores for task-specific compression of multi-task models.

Findings

Outperforms baseline pruning methods on six datasets.

Preserves performance in unseen domain settings.

Effective in low-resource and unsupervised scenarios.

Abstract

Multi-task language models show outstanding performance for various natural language understanding tasks with only a single model. However, these language models utilize an unnecessarily large number of model parameters, even when used only for a specific task. This paper proposes a novel training-free compression method for multi-task language models using a pruning method. Specifically, we use an attribution method to determine which neurons are essential for performing a specific task. We task-specifically prune unimportant neurons and leave only task-specific parameters. Furthermore, we extend our method to be applicable in low-resource and unsupervised settings. Since our compression method is training-free, it uses few computing resources and does not destroy the pre-trained knowledge of language models. Experimental results on the six widely-used datasets show that our proposed pruning method significantly outperforms baseline pruning methods. In addition, we demonstrate that our method preserves performance even in an unseen domain setting.