Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning

TL;DR

This paper introduces a general priming framework for parameter-efficient methods to improve few-shot adaptation and cross-task generalization of pre-trained language models across diverse NLP tasks.

Contribution

It proposes a novel PE priming framework that enhances few-shot learning and generalization capabilities of PLMs, with comprehensive evaluation on a large cross-domain benchmark.

Findings

Priming improves adaptation speed to target tasks.

Optimal priming strategies vary across tasks.

Priming significantly enhances cross-domain generalization.

Abstract

Parameter-efficient (PE) methods (like Prompts or Adapters) for adapting pre-trained language models (PLM) to downstream tasks have been popular recently. However, hindrances still prevent these methods from reaching their full potential. For example, two significant challenges are few-shot adaptation and cross-task generalization. To tackle these issues, we propose a general PE priming framework to enhance and explore the few-shot adaptation and generalization ability of PE methods. In this framework, PLMs are primed with PE methods for rapidly adapting to various target tasks. To evaluate the generalization ability of these PE methods, we conduct experiments on a few-shot cross-domain benchmark containing 160 diverse NLP tasks. Our experiment not only reveals the best priming strategy but also verifies that priming facilitates the adaptation to target tasks.