MerA: Merging Pretrained Adapters For Few-Shot Learning

TL;DR

MerA introduces an efficient method for merging pretrained adapters to improve few-shot learning performance, outperforming existing adapter-based methods and even full fine-tuning in some cases.

Contribution

The paper proposes MerA, a novel approach to merge pretrained adapters for better few-shot learning, with a new 'same-track' technique that further enhances performance.

Findings

MerA outperforms single adapters and AdapterFusion in few-shot tasks.

The 'same-track' merging technique yields significant performance gains.

MerA surpasses full fine-tuning and adapter tuning in several benchmarks.

Abstract

Adapter tuning, which updates only a few parameters, has become a mainstream method for fine-tuning pretrained language models to downstream tasks. However, it often yields subpar results in few-shot learning. AdapterFusion, which assembles pretrained adapters using composition layers tailored to specific tasks, is a possible solution but significantly increases trainable parameters and deployment costs. Despite this, our preliminary study reveals that even single adapters can outperform Adapterfusion in few-shot learning, urging us to propose \textbf{\texttt{Merging Pretrained Adapters}} (MerA) that efficiently incorporates pretrained adapters to a single model through model fusion. Extensive experiments on two PLMs demonstrate that MerA achieves substantial improvements compared to both single adapters and AdapterFusion. To further enhance the capacity of MerA, we also introduce a simple yet effective technique, referred to as the "\textit{same-track}" setting, that merges adapters from the same track of pretraining tasks. With the implementation of the "\textit{same-track}" setting, we observe even more impressive gains, surpassing the performance of both full fine-tuning and adapter tuning by a substantial margin, e.g., 3.5\% in MRPC and 5.0\% in MNLI.