A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models

TL;DR

This paper provides a comprehensive empirical comparison of tokenizer-free and subword-based multilingual pretrained models, highlighting practical considerations like memory, speed, and robustness alongside accuracy.

Contribution

It offers a multi-dimensional evaluation of tokenizer-free models, revealing that subword-based models often outperform in efficiency and practicality.

Findings

Subword-based models have lower inference latency.

Subword models use less memory.

Subword models often outperform tokenizer-free models in practical settings.

Abstract

Recent work on tokenizer-free multilingual pretrained models show promising results in improving cross-lingual transfer and reducing engineering overhead (Clark et al., 2022; Xue et al., 2022). However, these works mainly focus on reporting accuracy on a limited set of tasks and data settings, placing less emphasis on other important factors when tuning and deploying the models in practice, such as memory usage, inference speed, and fine-tuning data robustness. We attempt to fill this gap by performing a comprehensive empirical comparison of multilingual tokenizer-free and subword-based models considering these various dimensions. Surprisingly, we find that subword-based models might still be the most practical choice in many settings, achieving better performance for lower inference latency and memory usage. Based on these results, we encourage future work in tokenizer-free methods to consider these factors when designing and evaluating new models.