Meenz bleibt Meenz, but Large Language Models Do Not Speak Its Dialect

TL;DR

This paper investigates the capabilities of large language models to generate and understand the Meenzerisch dialect, revealing significant limitations and emphasizing the need for more resources and focused research on German dialects.

Contribution

First NLP study on Meenzerisch dialect, introducing a digital dictionary dataset and evaluating LLMs' ability to generate dialect words and definitions.

Findings

LLMs perform poorly in generating dialect definitions and words, with accuracy below 10%.

Few-shot learning and rule extraction improve results but remain insufficient.

Highlighting the need for more resources and dedicated research on German dialects.

Abstract

Meenzerisch, the dialect spoken in the German city of Mainz, is also the traditional language of the Mainz carnival, a yearly celebration well known throughout Germany. However, Meenzerisch is on the verge of dying out-a fate it shares with many other German dialects. Natural language processing (NLP) has the potential to help with the preservation and revival efforts of languages and dialects. However, so far no NLP research has looked at Meenzerisch. This work presents the first research in the field of NLP that is explicitly focused on the dialect of Mainz. We introduce a digital dictionary-an NLP-ready dataset derived from an existing resource (Schramm, 1966)-to support researchers in modeling and benchmarking the language. It contains 2,351 words in the dialect paired with their meanings described in Standard German. We then use this dataset to answer the following research questions: (1) Can state-of-the-art large language models (LLMs) generate definitions for dialect words? (2) Can LLMs generate words in Meenzerisch, given their definitions? Our experiments show that LLMs can do neither: the best model for definitions reaches only 6.27% accuracy and the best word generation model's accuracy is 1.51%. We then conduct two additional experiments in order to see if accuracy is improved by few-shot learning and by extracting rules from the training set, which are then passed to the LLM. While those approaches are able to improve the results, accuracy remains below 10%. This highlights that additional resources and an intensification of research efforts focused on German dialects are desperately needed.