Natural Language Processing for Dialects of a Language: A Survey

TL;DR

This survey reviews NLP research on dialects, highlighting datasets, approaches, and tasks across multiple languages, emphasizing the need for equitable language technologies and improved benchmarks.

Contribution

It provides a comprehensive overview of NLP methods for dialects, covering datasets, classical and deep learning approaches, and diverse tasks across languages.

Findings

NLP for dialects extends beyond classification to various NLU and NLG tasks.

Deep learning approaches, including pre-trained models, are increasingly used for dialect processing.

Research on dialects highlights challenges in creating equitable language technologies.

Abstract

State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectal datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches. We describe a wide range of NLP tasks in terms of two categories: natural language understanding (NLU) (for tasks such as dialect classification, sentiment analysis, parsing, and NLU benchmarks) and natural language generation (NLG) (for summarisation, machine translation, and dialogue systems). The survey is also broad in its coverage of languages which include English, Arabic, German, among others. We observe that past work in NLP concerning dialects goes deeper than mere dialect classification, and extends to several NLU and NLG tasks. For these tasks, we describe classical machine learning using statistical models, along with the recent deep learning-based approaches based on pre-trained language models. We expect that this survey will be useful to NLP researchers interested in building equitable language technologies by rethinking LLM benchmarks and model architectures.