ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws

TL;DR

ShapleyLaw introduces a game-theoretic framework for multilingual pretraining that accurately measures cross-lingual transfer effects, enabling better prediction of test loss and optimization of language mixture ratios.

Contribution

It proposes a novel cooperative game-theoretic approach, using Shapley values, to quantify cross-lingual transfer and improve multilingual scaling law predictions.

Findings

ShapleyLaw outperforms baseline methods in test loss prediction.

It effectively estimates optimal language mixture ratios.

The approach captures cross-lingual transfer effects more accurately.

Abstract

In multilingual pretraining, the test loss of a pretrained model is heavily influenced by the proportion of each language in the pretraining data, namely the \textit{language mixture ratios}. Multilingual scaling laws can predict the test loss under different language mixture ratios and can therefore be used to estimate the optimal ratios. However, the current approaches to multilingual scaling laws do not measure the \textit{cross-lingual transfer} effect, resulting in suboptimal mixture ratios. In this paper, we consider multilingual pretraining as a cooperative game in which each language acts as a player that jointly contributes to pretraining, gaining the resulting reduction in test loss as the payoff. Consequently, from the perspective of cooperative game theory, we quantify the cross-lingual transfer from each language by its contribution in the game, and propose a game-theoretic multilingual scaling law called \textit{ShapleyLaw}. Our experiments show that ShapleyLaw outperforms baseline methods in model performance prediction and language mixture optimization.