Morphological evaluation of subwords vocabulary used by BETO language model

TL;DR

This paper evaluates the morphological quality of subword vocabularies used by language models, revealing that BETO's vocabulary has low morphological quality and that larger training corpora do not improve it.

Contribution

It applies a novel morphological evaluation method to BETO's tokenizer, providing insights into subword quality and tokenizer algorithm identification.

Findings

BETO's vocabulary has low morphological quality.

Training on larger corpora does not improve morphological quality.

The evaluation clarifies the tokenizer algorithm used by BETO.

Abstract

Subword tokenization algorithms used by Large Language Models are significantly more efficient and can independently build the necessary vocabulary of words and subwords without human intervention. However, those subwords do not always align with real morphemes, potentially impacting the models' performance, though it remains uncertain when this might occur. In previous research, we proposed a method to assess the morphological quality of vocabularies, focusing on the overlap between these vocabularies and the morphemes of a given language. Our evaluation method was built on three quality measures, relevance, cohesion, and morphological accuracy, and a procedure for their assessment. By applying this method to vocabularies created by three subword tokenization algorithms, BPE, Wordpiece, and Unigram, we concluded that these vocabularies generally exhibit very low morphological quality. In this article, we apply this evaluation to the tokenizer of BETO, a BERT language model trained on large Spanish corpora. This evaluation, along with our previous results, helped us conclude that its vocabulary has a low morphological quality, and we also found that training the tokenizer in a larger corpus does not improve the morphological quality of the generated vocabulary. Additionally, this evaluation helps clarify the algorithm used by the tokenizer, that is, Wordpiece, given the inconsistencies between the authors' claims and the model's configuration.