User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization

TL;DR

This paper introduces a new Japanese user-generated text corpus for evaluating morphological analysis and lexical normalization, highlighting the challenges faced by current methods on non-standard language forms.

Contribution

The authors created a publicly available annotated corpus for Japanese UGT, providing a benchmark for future research in morphological analysis and normalization tasks.

Findings

Existing MA/LN methods perform poorly on non-standard forms.

The corpus reveals significant challenges in analyzing UGT.

Benchmark results indicate room for improvement in current systems.

Abstract

Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.