A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments

TL;DR

This paper introduces a subword embedding method for detecting linguistic variation in Luxembourgish user comments, capturing dialectal and stylistic differences without prior normalization or variant lists.

Contribution

The approach trains subword embeddings on raw text to identify variation patterns, providing a reproducible framework for low-resource language variation analysis.

Findings

Uncovered extensive lexical and orthographic variation

Captured systematic dialectal and stylistic patterns

Supported analysis with transparent clustering

Abstract

This paper presents an embedding-based approach to detecting variation without relying on prior normalisation or predefined variant lists. The method trains subword embeddings on raw text and groups related forms through combined cosine and n-gram similarity. This allows spelling and morphological diversity to be examined and analysed as linguistic structure rather than treated as noise. Using a large corpus of Luxembourgish user comments, the approach uncovers extensive lexical and orthographic variation that aligns with patterns described in dialectal and sociolinguistic research. The induced families capture systematic correspondences and highlight areas of regional and stylistic differentiation. The procedure does not strictly require manual annotation, but does produce transparent clusters that support both quantitative and qualitative analysis. The results demonstrate that distributional modelling can reveal meaningful patterns of variation even in ''noisy'' or low-resource settings, offering a reproducible methodological framework for studying language variety in multilingual and small-language contexts.