A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

TL;DR

This paper investigates the sensitivity of few-shot crosslingual transfer to shot selection, demonstrating the importance of standardization and showing that full model fine-tuning can outperform other methods.

Contribution

It highlights the high sensitivity of few-shot transfer to shot selection and provides a large-scale analysis across multiple languages and tasks, advocating for standardized experimental protocols.

Findings

Few-shot transfer performance varies significantly with shot choice.

Full model fine-tuning outperforms several state-of-the-art few-shot methods.

Providing sampled shots promotes standardization in future research.

Abstract

Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no attention has been paid to standardizing and analyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a large-scale experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. We provide an analysis of success and failure cases of few-shot transfer, which highlights the role of lexical features. Additionally, we show that a straightforward full model finetuning approach is quite effective for few-shot transfer, outperforming several state-of-the-art few-shot approaches. As a step towards standardizing few-shot crosslingual experimental designs, we make our sampled few shots publicly available.