Konooz: Multi-domain Multi-dialect Corpus for Named Entity Recognition

TL;DR

Konooz is a comprehensive multi-dialect, multi-domain Arabic corpus with extensive annotations, used to benchmark NER models and analyze cross-domain and cross-dialect performance issues in Arabic NLP.

Contribution

This paper introduces Konooz, a large, annotated multi-dialect Arabic corpus, and provides the first benchmarking of Arabic NER models across diverse domains and dialects.

Findings

NER model performance drops up to 38% across domains and dialects

Significant divergence observed between domains and dialects using MMD metric

Certain models perform better on specific dialects and domains

Abstract

We introduce Konooz, a novel multi-dimensional corpus covering 16 Arabic dialects across 10 domains, resulting in 160 distinct corpora. The corpus comprises about 777k tokens, carefully collected and manually annotated with 21 entity types using both nested and flat annotation schemes - using the Wojood guidelines. While Konooz is useful for various NLP tasks like domain adaptation and transfer learning, this paper primarily focuses on benchmarking existing Arabic Named Entity Recognition (NER) models, especially cross-domain and cross-dialect model performance. Our benchmarking of four Arabic NER models using Konooz reveals a significant drop in performance of up to 38% when compared to the in-distribution data. Furthermore, we present an in-depth analysis of domain and dialect divergence and the impact of resource scarcity. We also measured the overlap between domains and dialects using the Maximum Mean Discrepancy (MMD) metric, and illustrated why certain NER models perform better on specific dialects and domains. Konooz is open-source and publicly available at https://sina.birzeit.edu/wojood/#download