Exploiting Dialect Identification in Automatic Dialectal Text Normalization

TL;DR

This paper investigates how dialect identification can enhance the normalization of Dialectal Arabic into standard orthography, leveraging a unique corpus and pretrained models to improve NLP applications.

Contribution

It introduces a novel approach that incorporates dialect identification into Dialectal Arabic normalization, demonstrating improved performance across multiple dialects.

Findings

Dialect identification improves normalization accuracy

Pretrained sequence-to-sequence models outperform baselines

Publicly available code and data facilitate future research

Abstract

Dialectal Arabic is the primary spoken language used by native Arabic speakers in daily communication. The rise of social media platforms has notably expanded its use as a written language. However, Arabic dialects do not have standard orthographies. This, combined with the inherent noise in user-generated content on social media, presents a major challenge to NLP applications dealing with Dialectal Arabic. In this paper, we explore and report on the task of CODAfication, which aims to normalize Dialectal Arabic into the Conventional Orthography for Dialectal Arabic (CODA). We work with a unique parallel corpus of multiple Arabic dialects focusing on five major city dialects. We benchmark newly developed pretrained sequence-to-sequence models on the task of CODAfication. We further show that using dialect identification information improves the performance across all dialects. We make our code, data, and pretrained models publicly available.