The Impact of Model Scaling on Seen and Unseen Language Performance

TL;DR

This paper investigates how scaling multilingual large language models affects their performance on text classification and translation across 204 languages, revealing different scaling behaviors in zero-shot and few-shot settings.

Contribution

It provides a comprehensive analysis of multilingual LLM performance across many languages and model sizes, highlighting the impact of resource levels and task-specific scaling effects.

Findings

Zero-shot performance remains flat regardless of model size.

Larger models improve two-shot multilingual classification performance.

Only instruction-tuned models benefit from scaling in translation tasks.

Abstract

The rapid advancement of Large Language Models (LLMs), particularly those trained on multilingual corpora, has intensified the need for a deeper understanding of their performance across a diverse range of languages and model sizes. Our research addresses this critical need by studying the performance and scaling behavior of multilingual LLMs in text classification and machine translation tasks across 204 languages. We systematically examine both seen and unseen languages across three model families of varying sizes in zero-shot and few-shot settings. Our findings show significant differences in scaling behavior between zero-shot and two-shot scenarios, with striking disparities in performance between seen and unseen languages. Model scale has little effect on zero-shot performance, which remains mostly flat. However, in two-shot settings, larger models show clear linear improvements in multilingual text classification. For translation tasks, however, only the instruction-tuned model showed clear benefits from scaling. Our analysis also suggests that overall resource levels, not just the proportions of pretraining languages, are better predictors of model performance, shedding light on what drives multilingual LLM effectiveness.