Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

TL;DR

This study systematically evaluates various large language models for diacritic restoration in Romanian texts, revealing that model architecture, training data, and prompt design significantly influence performance, with GPT-4o achieving the highest accuracy.

Contribution

It provides a comprehensive comparison of LLMs for diacritic restoration in Romanian, highlighting factors affecting performance and guiding future NLP tool development for diacritic-rich languages.

Findings

GPT-4o achieves top diacritic restoration accuracy

Model architecture and training data impact performance variability

Prompt complexity influences model effectiveness

Abstract

Automatic diacritic restoration is crucial for text processing in languages with rich diacritical marks, such as Romanian. This study evaluates the performance of several large language models (LLMs) in restoring diacritics in Romanian texts. Using a comprehensive corpus, we tested models including OpenAI's GPT-3.5, GPT-4, GPT-4o, Google's Gemini 1.0 Pro, Meta's Llama 2 and Llama 3, MistralAI's Mixtral 8x7B Instruct, airoboros 70B, and OpenLLM-Ro's RoLlama 2 7B, under multiple prompt templates ranging from zero-shot to complex multi-shot instructions. Results show that models such as GPT-4o achieve high diacritic restoration accuracy, consistently surpassing a neutral echo baseline, while others, including Meta's Llama family, exhibit wider variability. These findings highlight the impact of model architecture, training data, and prompt design on diacritic restoration performance and outline promising directions for improving NLP tools for diacritic-rich languages.