Unsupervised Tokenization Learning

TL;DR

This study introduces an unsupervised tokenization method that leverages the 'transition freedom' metric, outperforming traditional statistical metrics across multiple languages, with performance influenced by language-specific adaptations and corpus size.

Contribution

The paper proposes a novel unsupervised tokenization approach based on 'transition freedom', demonstrating its effectiveness and language-specific adaptations compared to existing methods.

Findings

Transition freedom metric outperforms mutual information and conditional probability.

Different languages require different metric derivatives for optimal tokenization.

Larger corpora do not necessarily improve tokenization quality; model compression can enhance performance.

Abstract

In the presented study, we discover that the so-called "transition freedom" metric appears superior for unsupervised tokenization purposes in comparison to statistical metrics such as mutual information and conditional probability, providing F-measure scores in range from 0.71 to 1.0 across explored multilingual corpora. We find that different languages require different offshoots of that metric (such as derivative, variance, and "peak values") for successful tokenization. Larger training corpora do not necessarily result in better tokenization quality, while compressing the models by eliminating statistically weak evidence tends to improve performance. The proposed unsupervised tokenization technique provides quality better than or comparable to lexicon-based ones, depending on the language.