DAIL: Data Augmentation for In-Context Learning via Self-Paraphrase

TL;DR

DAIL enhances in-context learning by generating paraphrases with large language models and using voting to improve accuracy, especially in low-resource scenarios, without requiring high-quality demonstrations.

Contribution

This paper introduces DAIL, a novel data augmentation method that uses self-paraphrasing and voting to improve ICL performance in low-resource settings.

Findings

DAIL outperforms standard ICL and ensemble methods in low-resource scenarios.

Voting consistency can serve as a confidence measure without logits.

Self-paraphrasing improves robustness of in-context learning.

Abstract

In-Context Learning (ICL) combined with pre-trained large language models has achieved promising results on various NLP tasks. However, ICL requires high-quality annotated demonstrations which might not be available in real-world scenarios. To overcome this limitation, we propose \textbf{D}ata \textbf{A}ugmentation for \textbf{I}n-Context \textbf{L}earning (\textbf{DAIL}). DAIL leverages the intuition that large language models are more familiar with the content generated by themselves. It first utilizes the language model to generate paraphrases of the test sample and employs majority voting to determine the final result based on individual predictions. Our extensive empirical evaluation shows that DAIL outperforms the standard ICL method and other ensemble-based methods in the low-resource scenario. Additionally, we explore the use of voting consistency as a confidence score of the model when the logits of predictions are inaccessible. We believe our work will stimulate further research on ICL in low-resource settings.