PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning

TL;DR

PETapter is a novel approach that combines PEFT methods with PET-style classification heads, enhancing few-shot learning in NLP tasks efficiently and modularly, especially useful for specialized scientific domains.

Contribution

It introduces PETapter, a new method that improves few-shot NLP fine-tuning by integrating PET-style heads with parameter-efficient techniques, enabling better performance and modularity.

Findings

Achieves comparable performance to full PET fine-tuning.

Provides greater reliability and higher parameter efficiency.

Enables higher modularity and easy sharing of trained modules.

Abstract

Few-shot learning and parameter-efficient fine-tuning (PEFT) are crucial to overcome the challenges of data scarcity and ever growing language model sizes. This applies in particular to specialized scientific domains, where researchers might lack expertise and resources to fine-tune high-performing language models to nuanced tasks. We propose PETapter, a novel method that effectively combines PEFT methods with PET-style classification heads to boost few-shot learning capabilities without the significant computational overhead typically associated with full model training. We validate our approach on three established NLP benchmark datasets and one real-world dataset from communication research. We show that PETapter not only achieves comparable performance to full few-shot fine-tuning using pattern-exploiting training (PET), but also provides greater reliability and higher parameter efficiency while enabling higher modularity and easy sharing of the trained modules, which enables more researchers to utilize high-performing NLP-methods in their research.