Open Vocabulary Learning for Neural Chinese Pinyin IME

TL;DR

This paper introduces a neural P2C conversion model with an online updated vocabulary and sampling mechanism, improving Chinese input methods by better handling ambiguities and supporting open vocabulary learning during user input.

Contribution

It presents a novel neural P2C model with an online vocabulary update mechanism, enhancing input accuracy and adaptability over traditional fixed-vocabulary models.

Findings

Outperforms commercial IMEs and traditional models on standard datasets.

Online vocabulary updates improve user input following.

Supports open vocabulary learning during IME operation.

Abstract

Pinyin-to-character (P2C) conversion is the core component of pinyin-based Chinese input method engine (IME). However, the conversion is seriously compromised by the ambiguities of Chinese characters corresponding to pinyin as well as the predefined fixed vocabularies. To alleviate such inconveniences, we propose a neural P2C conversion model augmented by an online updated vocabulary with a sampling mechanism to support open vocabulary learning during IME working. Our experiments show that the proposed method outperforms commercial IMEs and state-of-the-art traditional models on standard corpus and true inputting history dataset in terms of multiple metrics and thus the online updated vocabulary indeed helps our IME effectively follows user inputting behavior.