Representations of Language Varieties Are Reliable Given Corpus Similarity Measures

TL;DR

This study evaluates whether digital geo-referenced corpora reliably represent local language varieties by analyzing similarity measures across multiple languages and sources, confirming their stability and consistency.

Contribution

It demonstrates that frequency-based corpus similarity measures reliably capture linguistic variation across diverse digital sources and language varieties.

Findings

High agreement between sources indicates reliable representation of language varieties.

Corpus similarity measures are stable across different languages and regions.

Digital corpora effectively model linguistic variation in geo-referenced data.

Abstract

This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the web and tweets), allowing us to evaluate whether such geo-referenced corpora are reliable for modelling linguistic variation. The basic idea is that, if each source adequately represents a single underlying language variety, then the similarity between these sources should be stable across all languages and countries. The paper shows that there is a consistent agreement between these sources using frequency-based corpus similarity measures. This provides further evidence that digital geo-referenced corpora consistently represent local language varieties.