Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings

TL;DR

This study evaluates the performance of large language models in low-resource South Asian languages, introducing new datasets and analyzing zero-shot learning capabilities, highlighting GPT-4's superior performance over other models.

Contribution

The paper provides new datasets for sentiment and hate speech in Bangla, Hindi, and Urdu, and offers a comprehensive comparison of LLMs in low-resource language tasks.

Findings

GPT-4 outperforms Llama 2 and Gemini across tasks.

English language consistently yields better performance.

Natural language inference shows the highest accuracy among tasks.

Abstract

Large language models (LLMs) have garnered significant interest in natural language processing (NLP), particularly their remarkable performance in various downstream tasks in resource-rich languages. Recent studies have highlighted the limitations of LLMs in low-resource languages, primarily focusing on binary classification tasks and giving minimal attention to South Asian languages. These limitations are primarily attributed to constraints such as dataset scarcity, computational costs, and research gaps specific to low-resource languages. To address this gap, we present datasets for sentiment and hate speech tasks by translating from English to Bangla, Hindi, and Urdu, facilitating research in low-resource language processing. Further, we comprehensively examine zero-shot learning using multiple LLMs in English and widely spoken South Asian languages. Our findings indicate that GPT-4 consistently outperforms Llama 2 and Gemini, with English consistently demonstrating superior performance across diverse tasks compared to low-resource languages. Furthermore, our analysis reveals that natural language inference (NLI) exhibits the highest performance among the evaluated tasks, with GPT-4 demonstrating superior capabilities.