Learning to Adapt to Low-Resource Paraphrase Generation

TL;DR

This paper introduces LAPA, a meta-learning based adapter for pre-trained language models that improves low-resource and domain adaptation in paraphrase generation, achieving state-of-the-art results with minimal data and parameters.

Contribution

LAPA is a novel three-stage training adapter that enables effective domain adaptation and low-resource paraphrase generation using meta-learning and minimal parameters.

Findings

Achieves state-of-the-art performance in low-resource settings

Requires only 2% of trainable parameters and 1% labeled data

Effective across supervised, unsupervised, and low-resource scenarios

Abstract

Paraphrase generation is a longstanding NLP task and achieves great success with the aid of large corpora. However, transferring a paraphrasing model to another domain encounters the problem of domain shifting especially when the data is sparse. At the same time, widely using large pre-trained language models (PLMs) faces the overfitting problem when training on scarce labeled data. To mitigate these two issues, we propose, LAPA, an effective adapter for PLMs optimized by meta-learning. LAPA has three-stage training on three types of related resources to solve this problem: 1. pre-training PLMs on unsupervised corpora, 2. inserting an adapter layer and meta-training on source domain labeled data, and 3. fine-tuning adapters on a small amount of target domain labeled data. This method enables paraphrase generation models to learn basic language knowledge first, then learn the paraphrasing task itself later, and finally adapt to the target task. Our experimental results demonstrate that LAPA achieves state-of-the-art in supervised, unsupervised, and low-resource settings on three benchmark datasets. With only 2\% of trainable parameters and 1\% labeled data of the target task, our approach can achieve a competitive performance with previous work.