Japanese Lexical Complexity for Non-Native Readers: A New Dataset

TL;DR

This paper introduces the first Japanese lexical complexity prediction dataset, enabling better understanding and annotation of complex words for non-native readers, with a BERT-based system demonstrating effective performance.

Contribution

It creates a novel Japanese LCP dataset with L1-specific scores and evaluates a BERT-based model for lexical complexity prediction.

Findings

BERT-based system effectively predicts Japanese lexical complexity.

Dataset includes L1-specific complexity scores.

First Japanese LCP dataset developed.

Abstract

Lexical complexity prediction (LCP) is the task of predicting the complexity of words in a text on a continuous scale. It plays a vital role in simplifying or annotating complex words to assist readers. To study lexical complexity in Japanese, we construct the first Japanese LCP dataset. Our dataset provides separate complexity scores for Chinese/Korean annotators and others to address the readers' L1-specific needs. In the baseline experiment, we demonstrate the effectiveness of a BERT-based system for Japanese LCP.