Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?

TL;DR

This study investigates how well multilingual language models understand semantic concepts at the subword level, revealing that shared subword semantics can improve cross-lingual transfer and model predictions.

Contribution

The paper introduces semantic tokens by merging semantically similar subwords and evaluates their impact on multilingual models across various tasks, highlighting the importance of subword semantics.

Findings

Shared subword semantics enhance model predictions across languages.

Semantic tokens improve zero-shot transfer performance.

Subword groups exhibit diverse semantic similarities, including synonyms and translations.

Abstract

Human understanding of text depends on general semantic concepts of words rather than their superficial forms. To what extent does our human intuition transfer to language models? In this work, we study the degree to which current multilingual language models (mLMs) understand based on subword-level semantic concepts. To this end, we form "semantic tokens" by merging the semantically similar subwords and their embeddings, and evaluate the updated mLMs on five heterogeneous multilingual downstream tasks. Results show that the general shared semantics could get the models a long way in making the predictions on mLMs with different tokenizers and model sizes. Inspections of the grouped subwords show that they exhibit a wide range of semantic similarities, including synonyms and translations across many languages and scripts. Lastly, we find that the zero-shot results with semantic tokens are on par with or even better than the original models on certain classification tasks, suggesting that the shared subword-level semantics may serve as the anchors for cross-lingual transfer.