Which Pieces Does Unigram Tokenization Really Need?

TL;DR

This paper clarifies the practical implementation of Unigram tokenization, introduces a simpler algorithm with comparable performance, and aims to enhance its adoption in NLP tasks.

Contribution

It provides a practical guide for Unigram tokenization implementation and proposes a simpler algorithm with marginally higher training loss for better compression.

Findings

A clear implementation guide for Unigram tokenization.

A new simpler algorithm with improved compression.

Potential for increased adoption of Unigram methods.

Abstract

The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding. Despite its theoretical elegance, its implementation in practice is complex, limiting its adoption to the SentencePiece package and adapters thereof. We bridge this gap between theory and practice by providing a clear guide to implementation and parameter choices. We also identify a simpler algorithm that accepts slightly higher training loss in exchange for improved compression.