"Diversity and Uncertainty in Moderation" are the Key to Data Selection for Multilingual Few-shot Transfer

TL;DR

This paper investigates data selection strategies based on diversity and uncertainty measures to improve few-shot transfer performance across multiple languages and tasks, demonstrating consistent gains over random selection.

Contribution

It introduces a novel loss embedding method for sequence labeling that enhances data selection by combining diversity and uncertainty, outperforming baseline methods.

Findings

Gradient and loss embedding strategies outperform random selection.

Methods show consistent improvements across POS, NER, and NLI tasks.

Performance gains vary with initial zero-shot transfer quality.

Abstract

Few-shot transfer often shows substantial gain over zero-shot transfer~\cite{lauscher2020zero}, which is a practically useful trade-off between fully supervised and unsupervised learning approaches for multilingual pretrained model-based systems. This paper explores various strategies for selecting data for annotation that can result in a better few-shot transfer. The proposed approaches rely on multiple measures such as data entropy using $n$-gram language model, predictive entropy, and gradient embedding. We propose a loss embedding method for sequence labeling tasks, which induces diversity and uncertainty sampling similar to gradient embedding. The proposed data selection strategies are evaluated and compared for POS tagging, NER, and NLI tasks for up to 20 languages. Our experiments show that the gradient and loss embedding-based strategies consistently outperform random data selection baselines, with gains varying with the initial performance of the zero-shot transfer. Furthermore, the proposed method shows similar trends in improvement even when the model is fine-tuned using a lower proportion of the original task-specific labeled training data for zero-shot transfer.