Which Encoding is the Best for Text Classification in Chinese, English, Japanese and Korean?

TL;DR

This study empirically compares various encoding methods for text classification across Chinese, English, Japanese, and Korean, analyzing their effectiveness with different models and datasets to identify optimal encoding strategies.

Contribution

It provides a comprehensive empirical comparison of encoding techniques for multilingual text classification, highlighting the effectiveness of byte-level, word-level, and character-level encodings across models.

Findings

Byte-level one-hot encoding with UTF-8 is consistently competitive for convolutional networks.

Word-level n-grams are effective even without perfect segmentation.

fastText performs best with character-level n-gram encoding but can overfit with rich features.

Abstract

This article offers an empirical study on the different ways of encoding Chinese, Japanese, Korean (CJK) and English languages for text classification. Different encoding levels are studied, including UTF-8 bytes, characters, words, romanized characters and romanized words. For all encoding levels, whenever applicable, we provide comparisons with linear models, fastText and convolutional networks. For convolutional networks, we compare between encoding mechanisms using character glyph images, one-hot (or one-of-n) encoding, and embedding. In total there are 473 models, using 14 large-scale text classification datasets in 4 languages including Chinese, English, Japanese and Korean. Some conclusions from these results include that byte-level one-hot encoding based on UTF-8 consistently produces competitive results for convolutional networks, that word-level n-grams linear models are competitive even without perfect word segmentation, and that fastText provides the best result using character-level n-gram encoding but can overfit when the features are overly rich.