MorphPiece : A Linguistic Tokenizer for Large Language Models

TL;DR

MorphPiece introduces a linguistically motivated tokenization method based on morphological segmentation, leading to a language model that outperforms GPT-2 on multiple NLP benchmarks with fewer training iterations.

Contribution

This paper presents MorphPiece, a novel morphological tokenization scheme that improves language model performance by incorporating linguistic features, unlike traditional statistical tokenizers.

Findings

MorphGPT outperforms GPT-2 on various NLP tasks.

MorphGPT requires about half the training iterations of GPT-2.

MorphPiece achieves superior results compared to FLOTA tokenization.

Abstract

Tokenization is a critical part of modern NLP pipelines. However, contemporary tokenizers for Large Language Models are based on statistical analysis of text corpora, without much consideration to the linguistic features. I propose a linguistically motivated tokenization scheme, MorphPiece, which is based partly on morphological segmentation of the underlying text. A GPT-style causal language model trained on this tokenizer (called MorphGPT) shows comparable or superior performance on a variety of supervised and unsupervised NLP tasks, compared to the OpenAI GPT-2 model. Specifically I evaluated MorphGPT on language modeling tasks, zero-shot performance on GLUE Benchmark with various prompt templates, massive text embedding benchmark (MTEB) for supervised and unsupervised performance, and lastly with another morphological tokenization scheme (FLOTA, Hoffmann et al., 2022) and find that the model trained on MorphPiece outperforms GPT-2 on most evaluations, at times with considerable margin, despite being trained for about half the training iterations.