Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

TL;DR

This study evaluates the effectiveness of seven leading LLMs in sentiment analysis within multilingual, code-mixed WhatsApp chats, highlighting their strengths and weaknesses in understanding cultural and linguistic nuances in low-resource scenarios.

Contribution

It provides a comprehensive evaluation of LLMs' performance in culturally nuanced, low-resource multilingual settings, emphasizing the need for improved benchmarks and transparency.

Findings

GPT-4 and GPT-4-Turbo excelled in understanding diverse linguistic and contextual nuances.

Most LLMs struggled with cultural nuances and transparency in decision-making.

High-performing models still face challenges in low-resource, culturally nuanced environments.

Abstract

The deployment of Large Language Models (LLMs) in real-world applications presents both opportunities and challenges, particularly in multilingual and code-mixed communication settings. This research evaluates the performance of seven leading LLMs in sentiment analysis on a dataset derived from multilingual and code-mixed WhatsApp chats, including Swahili, English and Sheng. Our evaluation includes both quantitative analysis using metrics like F1 score and qualitative assessment of LLMs' explanations for their predictions. We find that, while Mistral-7b and Mixtral-8x7b achieved high F1 scores, they and other LLMs such as GPT-3.5-Turbo, Llama-2-70b, and Gemma-7b struggled with understanding linguistic and contextual nuances, as well as lack of transparency in their decision-making process as observed from their explanations. In contrast, GPT-4 and GPT-4-Turbo excelled in grasping diverse linguistic inputs and managing various contextual information, demonstrating high consistency with human alignment and transparency in their decision-making process. The LLMs however, encountered difficulties in incorporating cultural nuance especially in non-English settings with GPT-4s doing so inconsistently. The findings emphasize the necessity of continuous improvement of LLMs to effectively tackle the challenges of culturally nuanced, low-resource real-world settings and the need for developing evaluation benchmarks for capturing these issues.