DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects

TL;DR

DIA-HARM introduces a comprehensive benchmark and dataset to evaluate the robustness of disinformation detection models across 50 English dialects, revealing significant vulnerabilities and disparities in current models.

Contribution

This work provides the first large-scale benchmark and corpus for assessing dialectal robustness in disinformation detection, highlighting systematic model vulnerabilities and transfer capabilities.

Findings

Human-written dialectal content reduces detection accuracy by 1.4-3.6% F1.

Fine-tuned transformers outperform zero-shot LLMs significantly.

Multilingual models generalize better across dialects than monolingual models.

Abstract

Harmful content detectors-particularly disinformation classifiers-are predominantly developed and evaluated on Standard American English (SAE), leaving their robustness to dialectal variation unexplored. We present DIA-HARM, the first benchmark for evaluating disinformation detection robustness across 50 English dialects spanning U.S., British, African, Caribbean, and Asia-Pacific varieties. Using Multi-VALUE's linguistically grounded transformations, we introduce D3 (Dialectal Disinformation Detection), a corpus of 195K samples derived from established disinformation benchmarks. Our evaluation of 16 detection models reveals systematic vulnerabilities: human-written dialectal content degrades detection by 1.4-3.6% F1, while AI-generated content remains stable. Fine-tuned transformers substantially outperform zero-shot LLMs (96.6% vs. 78.3% best-case F1), with some models exhibiting catastrophic failures exceeding 33% degradation on mixed content. Cross-dialectal transfer analysis across 2,450 dialect pairs shows that multilingual models (mDeBERTa: 97.2% average F1) generalize effectively, while monolingual models like RoBERTa and XLM-RoBERTa fail on dialectal inputs. These findings demonstrate that current disinformation detectors may systematically disadvantage hundreds of millions of non-SAE speakers worldwide. We release the DIA-HARM framework, D3 corpus, and evaluation tools: https://github.com/jsl5710/dia-harm