MEAL: Stable and Active Learning for Few-Shot Prompting

TL;DR

This paper introduces MEAL, a method combining ensembling and active learning to improve the stability and performance of few-shot prompt-based classification, addressing high variance issues.

Contribution

It proposes novel ensembling techniques and a new active learning criterion tailored for prompt-based few-shot learning, enhancing reliability and effectiveness.

Findings

Reduces run variability in few-shot learning

Improves prompt-based finetuning performance by 2.3 points

Provides publicly available code and data splits

Abstract

Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners. However, this approach has high variance both across different sets of few shots (data selection) and across different finetuning runs (run variability). This is problematic not only because it impedes the fair comparison of different approaches, but especially because it makes few-shot learning too unreliable for many real-world applications. To alleviate these issues, we make two contributions for more stable and effective few-shot learning: First, we propose novel ensembling methods and show that they substantially reduce run variability. Second, we introduce a new active learning (AL) criterion for data selection and present the first AL-based approach specifically tailored towards prompt-based learning. In our experiments, we show that our combined method, MEAL (Multiprompt finetuning and prediction Ensembling with Active Learning), improves overall performance of prompt-based finetuning by 2.3 points on five diverse tasks. We publicly share our code and data splits in https://github.com/akoksal/MEAL.