Diacritics Restoration using BERT with Analysis on Czech language

TL;DR

This paper introduces a BERT-based model for diacritics restoration across 12 languages, with a detailed error analysis on Czech, revealing that many mispredictions are plausible variants or data errors.

Contribution

The paper presents a novel BERT-based architecture for diacritics restoration and provides a comprehensive error analysis on Czech, highlighting the nature of mispredictions.

Findings

44% of mispredictions are not true errors

19% are plausible variants

25% are corrections of erroneous data

Abstract

We propose a new architecture for diacritics restoration based on contextualized embeddings, namely BERT, and we evaluate it on 12 languages with diacritics. Furthermore, we conduct a detailed error analysis on Czech, a morphologically rich language with a high level of diacritization. Notably, we manually annotate all mispredictions, showing that roughly 44% of them are actually not errors, but either plausible variants (19%), or the system corrections of erroneous data (25%). Finally, we categorize the real errors in detail. We release the code at https://github.com/ufal/bert-diacritics-restoration.