On Limitations of LLM as Annotator for Low Resource Languages

TL;DR

This paper evaluates the effectiveness of large language models as annotators for Marathi, a low-resource language, revealing their limitations compared to fine-tuned BERT models across various NLP classification tasks.

Contribution

It provides a comprehensive comparison of LLMs and BERT-based models for annotation in low-resource languages, highlighting current limitations of LLMs in this context.

Findings

LLMs perform poorly on Marathi compared to high-resource languages.

GPT-4o and Llama 3.1 405B underperform fine-tuned BERT by over 10%.

Results demonstrate significant gaps in LLM annotation capabilities for low-resource languages.

Abstract

Low-resource languages face significant challenges due to the lack of sufficient linguistic data, resources, and tools for tasks such as supervised learning, annotation, and classification. This shortage hinders the development of accurate models and datasets, making it difficult to perform critical NLP tasks like sentiment analysis or hate speech detection. To bridge this gap, Large Language Models (LLMs) present an opportunity for potential annotators, capable of generating datasets and resources for these underrepresented languages. In this paper, we focus on Marathi, a low-resource language, and evaluate the performance of both closed-source and open-source LLMs as annotators, while also comparing these results with fine-tuned BERT models. We assess models such as GPT-4o and Gemini 1.0 Pro, Gemma 2 (2B and 9B), and Llama 3.1 (8B and 405B) on classification tasks including sentiment analysis, news classification, and hate speech detection. Our findings reveal that while LLMs excel in annotation tasks for high-resource languages like English, they still fall short when applied to Marathi. Even advanced models like GPT-4o and Llama 3.1 405B underperform compared to fine-tuned BERT-based baselines, with GPT-4o and Llama 3.1 405B trailing fine-tuned BERT by accuracy margins of 10.2% and 14.1%, respectively. This highlights the limitations of LLMs as annotators for low-resource languages.