More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models

TL;DR

This paper investigates advanced tokenization methods, including statistical tests and encoding, to improve LDA topic modeling for languages without clear word boundaries, resulting in clearer and more coherent topics.

Contribution

It introduces new tokenization techniques for LDA that enhance topic quality in languages like Chinese and Thai, with a novel clustering metric for evaluation.

Findings

Merged tokens lead to clearer, more coherent topics

Statistical and encoding methods improve tokenization quality

Enhanced topic distinction in non-segmented languages

Abstract

Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection of documents to discover their latent topics using word-document co-occurrences. However, it is unclear how to achieve the best results for languages without marked word boundaries such as Chinese and Thai. Here, we explore the use of Pearson's chi-squared test, t-statistics, and Word Pair Encoding (WPE) to produce tokens as input to the LDA model. The Chi-squared, t, and WPE tokenizers are trained on Wikipedia text to look for words that should be grouped together, such as compound nouns, proper nouns, and complex event verbs. We propose a new metric for measuring the clustering quality in settings where the vocabularies of the models differ. Based on this metric and other established metrics, we show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those unmerged models.