DeformAr: Rethinking NER Evaluation through Component Analysis and Visual Analytics

TL;DR

DeformAr is a comprehensive framework combining data analysis and visual tools to diagnose and explain performance gaps in Arabic NER systems compared to English, addressing tokenisation, dataset quality, and annotation issues.

Contribution

It introduces DeformAr, the first Arabic-specific interpretability framework that analyzes data and model components jointly through interactive visualizations and diagnostic measures.

Findings

DeformAr effectively identifies factors affecting Arabic NER performance.

The framework reveals interactions between data quality and model behaviour.

It provides actionable insights for improving Arabic NER systems.

Abstract

Transformer models have significantly advanced Natural Language Processing (NLP), demonstrating strong performance in English. However, their effectiveness in Arabic, particularly for Named Entity Recognition (NER), remains limited, even with larger pre-trained models. This performance gap stems from multiple factors, including tokenisation, dataset quality, and annotation inconsistencies. Existing studies often analyze these issues in isolation, failing to capture their joint effect on system behaviour and performance. We introduce DeformAr (Debugging and Evaluation Framework for Transformer-based NER Systems), a novel framework designed to investigate and explain the performance discrepancy between Arabic and English NER systems. DeformAr integrates a data extraction library and an interactive dashboard, supporting two modes of evaluation: cross-component analysis and behavioural analysis. The framework divides each language into dataset and model components to examine their interactions. The analysis proceeds in two stages. First, cross-component analysis provides systematic diagnostic measures across data and model subcomponents, addressing the "what," "how," and "why" behind observed discrepancies. The second stage applies behavioural analysis by combining interpretability techniques with token-level metrics, interactive visualisations, and representation space analysis. These stages enable a component-aware diagnostic process that detects model behaviours and explains them by linking them to underlying representational patterns and data factors. DeformAr is the first Arabic-specific, component-based interpretability tool, offering a crucial resource for advancing model analysis in under-resourced languages.