Language Ideologies in a Multilingual Society: An LLM-based Analysis of Luxembourgish News Comments

TL;DR

This paper investigates the use of large language models to detect language ideologies in Luxembourgish news comments, highlighting challenges and potential in multilingual, low-resource settings.

Contribution

It evaluates LLM performance on ideological annotation tasks in Luxembourgish and explores translation to improve detection accuracy.

Findings

LLMs can identify language ideological content but are not yet fully optimized.

Translation to high-resource languages can improve LLM performance.

LLMs show promise as practical tools despite current limitations.

Abstract

Detecting language ideologies is a valuable yet complex task for understanding how identities are constructed through discourse. In Luxembourg's multicultural and multilingual society, language ideologies reflect more than simple preferences: they carry deep cultural and social meanings, shaping identities and social belonging. Following recent developments in applying Natural Language Processing tools to linguistics and social science, this paper explores the potential of large language models to assist in the detection of language ideologies. We manually annotate a corpus of user comments in Luxembourgish with predefined ideological categories and then evaluate the performance of large language models under varying prompt conditions to assess their ability to replicate these human annotations. Since Luxembourgish is a small language and poorly represented in the LLMs' training data, we also investigate whether machine-translating the data to high-resource languages increases performance on the ideology detection task. Our findings suggest that, while LLMs are not yet fully optimized for a multi-class ideological annotation task, they are practical tools to identify language ideological content.