Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning

TL;DR

This paper introduces a meta-learning approach to pretrain language models in a way that enhances the effectiveness of parameter-efficient fine-tuning methods, leading to improved performance on downstream tasks.

Contribution

It proposes a novel meta-learning framework using MAML to pretrain models specifically for parameter-efficient fine-tuning, which was not explored before.

Findings

Up to 1.7 point improvement on cross-lingual NER tasks.

Meta-learning tailored pretraining enhances fine-tuning efficiency.

Ablation studies confirm the importance of proposed modifications.

Abstract

A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer learning by updating only a small set of additional parameters while keeping the parameters of the pretrained language model frozen. While proven to be an effective method, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach should affect the pretraining stage. In this work, we show that taking the ultimate choice of fine-tuning method into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pretrained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 1.7 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the tweaks we introduce in MAML are crucial for the attained gains.