Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data

TL;DR

This paper introduces a method combining synthetic data generation and parameter-efficient fine-tuning to enhance low-resource large language model classification, achieving efficiency comparable to zero-shot methods and accuracy close to in-context learning.

Contribution

The paper presents a novel approach that uses synthetic data and PEFT to improve low-resource LLM classification without the high computational cost of ICL.

Findings

Achieves competitive accuracy with minimal data.

Reduces computational cost compared to ICL.

Effective across multiple datasets.

Abstract

Large Language Models (LLMs) operating in 0-shot or few-shot settings achieve competitive results in Text Classification tasks. In-Context Learning (ICL) typically achieves better accuracy than the 0-shot setting, but it pays in terms of efficiency, due to the longer input prompt. In this paper, we propose a strategy to make LLMs as efficient as 0-shot text classifiers, while getting comparable or better accuracy than ICL. Our solution targets the low resource setting, i.e., when only 4 examples per class are available. Using a single LLM and few-shot real data we perform a sequence of generation, filtering and Parameter-Efficient Fine-Tuning steps to create a robust and efficient classifier. Experimental results show that our approach leads to competitive results on multiple text classification datasets.