Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training

TL;DR

This paper introduces a task-agnostic mask training method to identify BERT subnetworks that better preserve pre-training performance and transferability, outperforming magnitude pruning in downstream tasks and data-scarce scenarios.

Contribution

It proposes a novel mask training approach that directly optimizes subnetworks for pre-training objectives, enhancing transferability and efficiency over traditional pruning methods.

Findings

Mask training improves downstream performance of BERT subnetworks.

The method is more efficient and effective in data-scarce fine-tuning scenarios.

Subnetworks found via mask training outperform those found by magnitude pruning.

Abstract

Recent studies on the lottery ticket hypothesis (LTH) show that pre-trained language models (PLMs) like BERT contain matching subnetworks that have similar transfer learning performance as the original PLM. These subnetworks are found using magnitude-based pruning. In this paper, we find that the BERT subnetworks have even more potential than these studies have shown. Firstly, we discover that the success of magnitude pruning can be attributed to the preserved pre-training performance, which correlates with the downstream transferability. Inspired by this, we propose to directly optimize the subnetwork structure towards the pre-training objectives, which can better preserve the pre-training performance. Specifically, we train binary masks over model weights on the pre-training tasks, with the aim of preserving the universal transferability of the subnetwork, which is agnostic to any specific downstream tasks. We then fine-tune the subnetworks on the GLUE benchmark and the SQuAD dataset. The results show that, compared with magnitude pruning, mask training can effectively find BERT subnetworks with improved overall performance on downstream tasks. Moreover, our method is also more efficient in searching subnetworks and more advantageous when fine-tuning within a certain range of data scarcity. Our code is available at https://github.com/llyx97/TAMT.