How Well Do LLMs Understand Tunisian Arabic?

TL;DR

This paper evaluates how well large language models understand Tunisian Arabic, highlighting gaps and emphasizing the need for inclusive AI that supports low-resource languages.

Contribution

Introduces a new dataset with Tunisian Arabic and English, benchmarking LLMs on transliteration, translation, and sentiment analysis tasks.

Findings

Significant variation in model performance across tasks

Identified limitations in LLM understanding of Tunisian dialects

Highlighted the importance of supporting low-resource languages in AI

Abstract

Large Language Models (LLMs) are the engines driving today's AI agents. The better these models understand human languages, the more natural and user-friendly the interaction with AI becomes, from everyday devices like computers and smartwatches to any tool that can act intelligently. Yet, the ability of industrial-scale LLMs to comprehend low-resource languages, such as Tunisian Arabic (Tunizi), is often overlooked. This neglect risks excluding millions of Tunisians from fully interacting with AI in their own language, pushing them toward French or English. Such a shift not only threatens the preservation of the Tunisian dialect but may also create challenges for literacy and influence younger generations to favor foreign languages. In this study, we introduce a novel dataset containing parallel Tunizi, standard Tunisian Arabic, and English translations, along with sentiment labels. We benchmark several popular LLMs on three tasks: transliteration, translation, and sentiment analysis. Our results reveal significant differences between models, highlighting both their strengths and limitations in understanding and processing Tunisian dialects. By quantifying these gaps, this work underscores the importance of including low-resource languages in the next generation of AI systems, ensuring technology remains accessible, inclusive, and culturally grounded.