Large Language Models for Sentiment Analysis to Detect Social Challenges: A Use Case with South African Languages

TL;DR

This study evaluates the zero-shot sentiment analysis capabilities of various large language models on social media posts in South African languages, highlighting the potential for social challenge detection and improved decision-making.

Contribution

It investigates the performance of multiple LLMs on multilingual sentiment analysis in South African languages, demonstrating that model fusion significantly enhances accuracy.

Findings

Significant performance differences among LLMs across languages and topics.

Fusion of LLM outcomes reduces sentiment classification errors below 1%.

Feasibility of reliable sentiment analysis systems for social challenge detection.

Abstract

Sentiment analysis can aid in understanding people's opinions and emotions on social issues. In multilingual communities sentiment analysis systems can be used to quickly identify social challenges in social media posts, enabling government departments to detect and address these issues more precisely and effectively. Recently, large-language models (LLMs) have become available to the wide public and initial analyses have shown that they exhibit magnificent zero-shot sentiment analysis abilities in English. However, there is no work that has investigated to leverage LLMs for sentiment analysis on social media posts in South African languages and detect social challenges. Consequently, in this work, we analyse the zero-shot performance of the state-of-the-art LLMs GPT-3.5, GPT-4, LlaMa 2, PaLM 2, and Dolly 2 to investigate the sentiment polarities of the 10 most emerging topics in English, Sepedi and Setswana social media posts that fall within the jurisdictional areas of 10 South African government departments. Our results demonstrate that there are big differences between the various LLMs, topics, and languages. In addition, we show that a fusion of the outcomes of different LLMs provides large gains in sentiment classification performance with sentiment classification errors below 1%. Consequently, it is now feasible to provide systems that generate reliable information about sentiment analysis to detect social challenges and draw conclusions about possible needs for actions on specific topics and within different language groups.