\'UFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5

TL;DR

This paper describes a multilingual lexical normalization system that leverages fine-tuned ByT5 models, achieving state-of-the-art results on social media datasets across 11 languages, validated through intrinsic and extrinsic evaluations.

Contribution

The authors introduce a novel approach using fine-tuned ByT5 models for multilingual lexical normalization, significantly improving performance over existing methods.

Findings

Achieved top performance in lexical normalization across 12 datasets in 11 languages.

Outperformed previous systems in both intrinsic and extrinsic evaluations.

Open-sourced the code and models for community use.

Abstract

We present the winning entry to the Multilingual Lexical Normalization (MultiLexNorm) shared task at W-NUT 2021 (van der Goot et al., 2021a), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages. We base our solution on a pre-trained byte-level language model, ByT5 (Xue et al., 2021a), which we further pre-train on synthetic data and then fine-tune on authentic normalization data. Our system achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. The source code is released at https://github.com/ufal/multilexnorm2021 and the fine-tuned models at https://huggingface.co/ufal.