Correcting diacritics and typos with a ByT5 transformer model

TL;DR

This paper presents a universal byte-level transformer model that simultaneously restores diacritics and corrects typos across multiple languages, outperforming traditional methods and demonstrating high accuracy with less data.

Contribution

The study introduces a single ByT5 transformer approach for combined diacritics restoration and typos correction, applicable to multiple languages without language-specific modifications.

Findings

Achieves over 98% accuracy in diacritics restoration on benchmark datasets.

Restores diacritics in unseen words with over 76% accuracy.

Reaches over 94% alpha-word accuracy in combined diacritics and typo correction across 13 languages.

Abstract

Due to the fast pace of life and online communications and the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing in other languages. Restoring diacritics and correcting spelling is important for proper language use and the disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately: the state-of-the-art diacritics restoration methods do not tolerate other typos, but classical spellcheckers also cannot deal adequately with all the diacritics missing. In this work, we tackle both problems at once by employing the newly-developed universal ByT5 byte-level seq2seq transformer model that requires no language-specific model structures. For a comparison, we perform diacritics restoration on benchmark datasets of 12 languages, with the addition of Lithuanian. The experimental investigation proves that our approach is able to achieve results (> 98%) comparable to the previous state-of-the-art, despite being trained less and on fewer data. Our approach is also able to restore diacritics in words not seen during training with > 76% accuracy. Our simultaneous diacritics restoration and typos correction approach reaches > 94% alpha-word accuracy on the 13 languages. It has no direct competitors and strongly outperforms classical spell-checking or dictionary-based approaches. We also demonstrate all the accuracies to further improve with more training. Taken together, this shows the great real-world application potential of our suggested methods to more data, languages, and error classes.