MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages

TL;DR

This paper extends the MultiLexNorm benchmark to include five Asian languages with diverse scripts, introduces a new LLM-based model for lexical normalization, and demonstrates its improved robustness over previous models.

Contribution

It expands the benchmark to Asian languages and proposes a novel LLM-based model for more effective lexical normalization across diverse scripts.

Findings

The new model outperforms previous state-of-the-art on Asian languages.

Performance drops when applying previous models to Asian languages.

Analysis highlights future research directions for lexical normalization.

Abstract

Social media data has been of interest to Natural Language Processing (NLP) practitioners for over a decade, because of its richness in information, but also challenges for automatic processing. Since language use is more informal, spontaneous, and adheres to many different sociolects, the performance of NLP models often deteriorates. One solution to this problem is to transform data to a standard variant before processing it, which is also called lexical normalization. There has been a wide variety of benchmarks and models proposed for this task. The MultiLexNorm benchmark proposed to unify these efforts, but it consists almost solely of languages from the Indo-European language family in the Latin script. Hence, we propose an extension to MultiLexNorm, which covers 5 Asian languages from different language families in 4 different scripts. We show that the previous state-of-the-art model performs worse on the new languages and propose a new architecture based on Large Language Models (LLMs), which shows more robust performance. Finally, we analyze remaining errors, revealing future directions for this task.