Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning

TL;DR

This paper investigates how instruction-tuning affects the calibration of multilingual language models, revealing that while confidence increases in low-resource languages, accuracy gains are limited, and label smoothing can improve calibration without additional data.

Contribution

It provides the first comprehensive analysis of multilingual calibration effects of instruction-tuning on LLMs, highlighting challenges and solutions for better reliability across languages.

Findings

Confidence increases in low-resource languages after instruction-tuning.

Accuracy improvements are marginal or non-existent in low-resource languages.

Label smoothing helps maintain better calibration without extra data.

Abstract

Ensuring that deep learning models are well-calibrated in terms of their predictive uncertainty is essential in maintaining their trustworthiness and reliability, yet despite increasing advances in foundation model research, the relationship between such large language models (LLMs) and their calibration remains an open area of research. In this work, we look at a critical gap in the calibration of LLMs within multilingual settings, in an attempt to better understand how the data scarcity can potentially lead to different calibration effects and how commonly used techniques can apply in these settings. Our analysis on two multilingual benchmarks, over 29 and 42 languages respectively, reveals that even in low-resource languages, model confidence can increase significantly after instruction-tuning on high-resource language SFT datasets. However, improvements in accuracy are marginal or non-existent, resulting in mis-calibration, highlighting a critical shortcoming of standard SFT for multilingual languages. Furthermore, we observe that the use of label smoothing to be a reasonable method alleviate this concern, again without any need for low-resource SFT data, maintaining better calibration across all languages. Overall, this highlights the importance of multilingual considerations for both training and tuning LLMs in order to improve their reliability and fairness in downstream use.