On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model

TL;DR

This paper compares prompt tuning and fine-tuning on multilingual autoregressive models, finding prompt tuning often outperforms fine-tuning, especially for low-resource languages, with minimal parameter updates.

Contribution

It provides the first comprehensive comparison of prompt tuning versus fine-tuning on decoder-based multilingual models across multiple cross-lingual tasks.

Findings

Prompt tuning matches or exceeds fine-tuning performance.

Prompt tuning updates only 0.13% of model parameters.

Prompt tuning is more effective for low-resource languages.

Abstract

An exciting advancement in the field of multilingual models is the emergence of autoregressive models with zero- and few-shot capabilities, a phenomenon widely reported in large-scale language models. To further improve model adaptation to cross-lingual tasks, another trend is to further fine-tune the language models with either full fine-tuning or parameter-efficient tuning. However, the interaction between parameter-efficient fine-tuning (PEFT) and cross-lingual tasks in multilingual autoregressive models has yet to be studied. Specifically, we lack an understanding of the role of linguistic distributions in multilingual models in the effectiveness of token-based prompt tuning. To address this question, we conduct experiments comparing prompt tuning and fine-tuning on the decoder-based multilingual model, XGLM, with four cross-lingual tasks (XNLI, PAWS-X, POS, NER). According to our study, prompt tuning achieves on par or better performance over fine-tuning across all languages while updating at most 0.13\% of the model parameters. Moreover, we empirically show that prompt tuning is more effective in enhancing the performance of low-resource languages than fine-tuning. Our further analysis shows that the phenomenon is related to the tokenization scheme of the multilingual model.