Learning to Recognize Dialect Features

TL;DR

This paper introduces a multitask learning approach using pretrained transformers to detect dialect features in speech and text, demonstrating high accuracy with minimal training data and applications in dialect classification.

Contribution

It presents the task of dialect feature detection and develops two multitask transformer-based models that effectively recognize dialect features with limited annotated data.

Findings

Models achieve high accuracy on Indian English dialect features.

Few minimal pairs suffice for effective training.

Dialect feature detection aids in dialect density measurement and classification.

Abstract

Building NLP systems that serve everyone requires accounting for dialect differences. But dialects are not monolithic entities: rather, distinctions between and within dialects are captured by the presence, absence, and frequency of dozens of dialect features in speech and text, such as the deletion of the copula in "He {} running". In this paper, we introduce the task of dialect feature detection, and present two multitask learning approaches, both based on pretrained transformers. For most dialects, large-scale annotated corpora for these features are unavailable, making it difficult to train recognizers. We train our models on a small number of minimal pairs, building on how linguists typically define dialect features. Evaluation on a test set of 22 dialect features of Indian English demonstrates that these models learn to recognize many features with high accuracy, and that a few minimal pairs can be as effective for training as thousands of labeled examples. We also demonstrate the downstream applicability of dialect feature detection both as a measure of dialect density and as a dialect classifier.