The Evolution of Darija Open Dataset: Introducing Version 2

TL;DR

This paper introduces Darija Open Dataset Version 2, a large, annotated, and multilingual dataset designed to advance NLP research and applications for the Moroccan dialect Darija, supporting linguistic diversity and technological development.

Contribution

The paper presents the creation and features of DODa V2, the largest open-source Darija-English dataset with rich annotations and multilingual entries, enhancing NLP resources for underrepresented dialects.

Findings

DODa V2 contains approximately 100,000 entries.

The dataset includes semantic and syntactic annotations.

It supports both Latin and Arabic scripts.

Abstract

Darija Open Dataset (DODa) represents an open-source project aimed at enhancing Natural Language Processing capabilities for the Moroccan dialect, Darija. With approximately 100,000 entries, DODa stands as the largest collaborative project of its kind for Darija-English translation. The dataset features semantic and syntactic categorizations, variations in spelling, verb conjugations across multiple tenses, as well as tens of thousands of translated sentences. The dataset includes entries written in both Latin and Arabic alphabets, reflecting the linguistic variations and preferences found in different sources and applications. The availability of such dataset is critical for developing applications that can accurately understand and generate Darija, thus supporting the linguistic needs of the Moroccan community and potentially extending to similar dialects in neighboring regions. This paper explores the strategic importance of DODa, its current achievements, and the envisioned future enhancements that will continue to promote its use and expansion in the global NLP landscape.