LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning

TL;DR

This paper introduces LM-CPPF, a novel data augmentation technique using paraphrasing with large language models to improve contrastive prompt-based fine-tuning on small NLP datasets.

Contribution

It proposes a new paraphrasing-guided data augmentation method for contrastive prompt-based fine-tuning using large language models like GPT-3 and OPT-175B.

Findings

Outperforms existing augmentation methods such as back translation and easy data augmentation.

Enhances model performance on multiple text classification benchmarks.

Demonstrates effectiveness of paraphrasing-guided augmentation in few-shot learning scenarios.

Abstract

In recent years, there has been significant progress in developing pre-trained language models for NLP. However, these models often struggle when fine-tuned on small datasets. To address this issue, researchers have proposed various adaptation approaches. Prompt-based tuning is arguably the most common way, especially for larger models. Previous research shows that adding contrastive learning to prompt-based fine-tuning is effective as it helps the model generate embeddings that are more distinguishable between classes, and it can also be more sample-efficient as the model learns from positive and negative examples simultaneously. One of the most important components of contrastive learning is data augmentation, but unlike computer vision, effective data augmentation for NLP is still challenging. This paper proposes LM-CPPF, Contrastive Paraphrasing-guided Prompt-based Fine-tuning of Language Models, which leverages prompt-based few-shot paraphrasing using generative language models, especially large language models such as GPT-3 and OPT-175B, for data augmentation. Our experiments on multiple text classification benchmarks show that this augmentation method outperforms other methods, such as easy data augmentation, back translation, and multiple templates.