Language Models are Few-shot Multilingual Learners

TL;DR

This paper demonstrates that large pre-trained language models like GPT and T5 can perform few-shot classification tasks across multiple languages without additional training, achieving competitive results in cross-lingual settings.

Contribution

It provides empirical evidence that multilingual capabilities emerge in large language models through few-shot learning, without explicit multilingual training.

Findings

Models can classify non-English samples using English examples as context.

Few-shot cross-lingual performance surpasses random chance.

Results are competitive with specialized cross-lingual models.

Abstract

General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (NLP) tasks and benchmarks when inferring instructions from very few examples. Here, we evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages without any parameter updates. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones. Finally, we find the in-context few-shot cross-lingual prediction results of language models are significantly better than random prediction, and they are competitive compared to the existing state-of-the-art cross-lingual models.