Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages

TL;DR

This paper presents a comprehensive evaluation of lexical normalization methods for unsegmented languages, introducing a large Japanese dataset and assessing state-of-the-art models across multiple metrics.

Contribution

It introduces a large-scale Japanese normalization dataset and evaluates multiple normalization approaches using diverse evaluation perspectives.

Findings

Encoder-only and decoder-only models achieve high accuracy.

Pretrained models are effective for lexical normalization.

Multi-domain evaluation provides comprehensive insights.

Abstract

Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text, yet the absence of comprehensive evaluations leaves it unclear which methods excel across multiple perspectives. Focusing on unsegmented languages, we make three key contributions: (1) creating a large-scale, multi-domain Japanese normalization dataset, (2) developing normalization methods based on state-of-the-art pretrained models, and (3) conducting experiments across multiple evaluation perspectives. Our experiments show that both encoder-only and decoder-only approaches achieve promising results in both accuracy and efficiency.