Statistical Augmentation of a Chinese Machine-Readable Dictionary

TL;DR

This paper presents a statistical method to enhance a Chinese dictionary by extracting domain-specific and regional words from a corpus, improving coverage and performance in Chinese tokenization.

Contribution

The paper introduces a novel statistical approach for augmenting Chinese dictionaries with new words, including idioms, names, and technical terms, not present in existing resources.

Findings

Enhanced dictionary coverage with new words

Improved Chinese tokenization accuracy

Validated approach with human and automated evaluation

Abstract

We describe a method of using statistically-collected Chinese character groups from a corpus to augment a Chinese dictionary. The method is particularly useful for extracting domain-specific and regional words not readily available in machine-readable dictionaries. Output was evaluated both using human evaluators and against a previously available dictionary. We also evaluated performance improvement in automatic Chinese tokenization. Results show that our method outputs legitimate words, acronymic constructions, idioms, names and titles, as well as technical compounds, many of which were lacking from the original dictionary.