NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities

TL;DR

This paper presents NileChat, a culturally aware LLM for Egyptian and Moroccan dialects, created through community-specific synthetic and retrieval-based data, outperforming similar-sized models in understanding, translation, and cultural alignment.

Contribution

It introduces a novel methodology for culturally tailored pre-training data and develops NileChat, an LLM that incorporates language, heritage, and values of low-resource communities.

Findings

NileChat outperforms existing Arabic-aware LLMs of similar size.

NileChat performs on par with larger models in various benchmarks.

The methodology enhances cultural and values alignment in LLMs.

Abstract

Enhancing the linguistic capabilities of Large Language Models (LLMs) to include low-resource languages is a critical research area. Current research directions predominantly rely on synthetic data generated by translating English corpora, which, while demonstrating promising linguistic understanding and translation abilities, often results in models aligned with source language culture. These models frequently fail to represent the cultural heritage and values of local communities. This work proposes a methodology to create both synthetic and retrieval-based pre-training data tailored to a specific community, considering its (i) language, (ii) cultural heritage, and (iii) cultural values. We demonstrate our methodology using Egyptian and Moroccan dialects as testbeds, chosen for their linguistic and cultural richness and current underrepresentation in LLMs. As a proof-of-concept, we develop NileChat, a 3B parameter Egyptian and Moroccan Arabic LLM adapted for Egyptian and Moroccan communities, incorporating their language, cultural heritage, and values. Our results on various understanding, translation, and cultural and values alignment benchmarks show that NileChat outperforms existing Arabic-aware LLMs of similar size and performs on par with larger models. This work addresses Arabic dialect in LLMs with a focus on cultural and values alignment via controlled synthetic data generation and retrieval-augmented pre-training for Moroccan Darija and Egyptian Arabic, including Arabizi variants, advancing Arabic NLP for low-resource communities. We share our methods, data, and models with the community to promote the inclusion and coverage of more diverse communities in cultural LLM development: https://github.com/UBC-NLP/nilechat .