LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages

TL;DR

This paper introduces a method using large language models within an active learning framework to efficiently annotate data for low-resource languages, significantly reducing costs and improving NLP capabilities in underrepresented languages.

Contribution

It proposes leveraging LLMs like GPT-4-Turbo in active learning for low-resource languages, achieving high performance with minimal data and cost savings.

Findings

Near-state-of-the-art performance with less data

Estimated cost savings of over 42 times compared to human annotation

Effective integration of LLMs in active learning for low-resource NLP

Abstract

Low-resource languages face significant barriers in AI development due to limited linguistic resources and expertise for data labeling, rendering them rare and costly. The scarcity of data and the absence of preexisting tools exacerbate these challenges, especially since these languages may not be adequately represented in various NLP datasets. To address this gap, we propose leveraging the potential of LLMs in the active learning loop for data annotation. Initially, we conduct evaluations to assess inter-annotator agreement and consistency, facilitating the selection of a suitable LLM annotator. The chosen annotator is then integrated into a training loop for a classifier using an active learning paradigm, minimizing the amount of queried data required. Empirical evaluations, notably employing GPT-4-Turbo, demonstrate near-state-of-the-art performance with significantly reduced data requirements, as indicated by estimated potential cost savings of at least 42.45 times compared to human annotation. Our proposed solution shows promising potential to substantially reduce both the monetary and computational costs associated with automation in low-resource settings. By bridging the gap between low-resource languages and AI, this approach fosters broader inclusion and shows the potential to enable automation across diverse linguistic landscapes.