To token or not to token: A Comparative Study of Text Representations for Cross-Lingual Transfer

TL;DR

This study compares different text representation models for cross-lingual transfer, analyzing their performance across multiple languages and tasks to guide model selection based on language similarity and task type.

Contribution

It introduces the Language Quotient (LQ) metric for evaluating cross-lingual transfer and provides a comprehensive comparison of segmentation-based, image-based, and character-level models.

Findings

Image-based models excel with closely related languages and similar scripts.

Segmentation-based models perform better on meaning-focused tasks like POS and NER.

Character-level models outperform others in dependency parsing tasks.

Abstract

Choosing an appropriate tokenization scheme is often a bottleneck in low-resource cross-lingual transfer. To understand the downstream implications of text representation choices, we perform a comparative analysis on language models having diverse text representation modalities including 2 segmentation-based models (\texttt{BERT}, \texttt{mBERT}), 1 image-based model (\texttt{PIXEL}), and 1 character-level model (\texttt{CANINE}). First, we propose a scoring Language Quotient (LQ) metric capable of providing a weighted representation of both zero-shot and few-shot evaluation combined. Utilizing this metric, we perform experiments comprising 19 source languages and 133 target languages on three tasks (POS tagging, Dependency parsing, and NER). Our analysis reveals that image-based models excel in cross-lingual transfer when languages are closely related and share visually similar scripts. However, for tasks biased toward word meaning (POS, NER), segmentation-based models prove to be superior. Furthermore, in dependency parsing tasks where word relationships play a crucial role, models with their character-level focus, outperform others. Finally, we propose a recommendation scheme based on our findings to guide model selection according to task and language requirements.