One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

TL;DR

This paper introduces PROPETL, a parameter-efficient transfer learning method that shares a single prototype network across tasks and layers, using binary masks to select sub-networks, significantly reducing storage while maintaining performance.

Contribution

Proposes PROPETL, a novel PETL approach that shares a prototype network across tasks and layers with binary masks for sub-network selection, enhancing efficiency.

Findings

Outperforms existing PETL methods in various tasks.

Uses approximately 10% of the parameter storage of previous methods.

Demonstrates the effectiveness of binary masks in identifying crucial network information.

Abstract

Fine-tuning pre-trained language models for multiple tasks tends to be expensive in terms of storage. To mitigate this, parameter-efficient transfer learning (PETL) methods have been proposed to address this issue, but they still require a significant number of parameters and storage when being applied to broader ranges of tasks. To achieve even greater storage reduction, we propose PROPETL, a novel method that enables efficient sharing of a single PETL module which we call prototype network (e.g., adapter, LoRA, and prefix-tuning) across layers and tasks. We then learn binary masks to select different sub-networks from the shared prototype network and apply them as PETL modules into different layers. We find that the binary masks can determine crucial information from the network, which is often ignored in previous studies. Our work can also be seen as a type of pruning method, where we find that overparameterization also exists in the seemingly small PETL modules. We evaluate PROPETL on various downstream tasks and show that it can outperform other PETL methods with approximately 10% of the parameter storage required by the latter.