Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration

TL;DR

This paper introduces probability calibration techniques to enhance zero-shot performance of multilingual encoders by correcting biases toward frequent label words, leading to significant improvements across various tasks.

Contribution

The paper proposes and validates calibration methods that improve zero-shot and few-shot multilingual encoder performance without retraining the models.

Findings

Calibration improves label word probability estimates.

Performance gains observed across multiple multilingual tasks.

Effective in both monolingual and multilingual settings.

Abstract

Pretrained multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts. This is accomplished by predicting the probabilities of the label words at the masked token position, without requiring any updates to the model parameters. However, the performance of this method is limited by the model's bias toward predicting label words which frequently occurred during the pretraining. These words typically receive high probabilities. To address this issue, we combine the models with calibration techniques which modify the probabilities of label words predicted by the models. We first validate the effectiveness of a proposed simple calibration method together with other existing techniques on monolingual encoders in both zero- and few-shot scenarios. We subsequently employ these calibration techniques on multilingual encoders, resulting in substantial performance improvements across a wide range of tasks.