DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English

TL;DR

DialectLLM introduces a large-scale, dialect-aware dialog dataset and benchmark, improving the authenticity of non-SAE dialect generation and highlighting current model limitations in dialect identification.

Contribution

The paper presents DialectLLM, a novel framework for creating authentic multi-dialectal dialog data and a benchmark for evaluating dialect understanding in LLMs.

Findings

Human evaluators prefer DialectLLM data over prior methods in 98.8% of comparisons.

Current LLMs achieve under 70% accuracy in dialect identification.

Models often misclassify non-SAE dialects as American or British.

Abstract

More than 80% of the 1.6B English speakers do not use Standard American English (SAE), yet LLMs often fail to correctly identify non-SAE dialects and generate stereotyped responses for their speakers. We introduce DialectLLM, the first large-scale framework for generating high-quality multi-dialectal conversational data encompassing the three pillars of written dialect -- lexical (vocabulary), orthographic (spelling), and morphosyntactic (grammar) features. DialectLLM produces a dialect-parallel dialog dataset spanning nine English dialects. Partnering with native linguists, we design and validate SAE-to-dialect transformation rules, ensuring authenticity. Our approach challenges the prevailing practice of applying a single morphosyntactic feature set to both user utterances and model responses, showing that models should not reproduce up to 90% of the grammatical features of a dialect. Human evaluation confirms data quality, with annotators preferring DialectLLM over prior methods in 98.8% of pairwise comparisons for dialect naturalness. We then construct DialectLLM-Bench, a dialect-parallel benchmark with 50k+ dialogs, resulting in 97k+ QA pairs, and evaluate 17 LLMs on dialect identification and response generation tasks. Even frontier models achieve under 70% accuracy, fail to reach 50% for prominent dialects like Canadian English, and systematically misclassify non-SAE dialects as American or British. Beyond benchmarking, we show that DialectLLM data also serve as a scalable LLM post-training resource, suggesting a practical path toward dialect-aware conversational AI.