Consistency Regularization for Cross-Lingual Fine-Tuning

TL;DR

This paper introduces a consistency regularization approach to enhance cross-lingual fine-tuning of pre-trained language models, improving performance across multiple NLP tasks by penalizing prediction sensitivity to data augmentations and model variations.

Contribution

The work proposes a novel consistency regularization method that leverages data and model consistency to improve cross-lingual transfer in NLP tasks.

Findings

Significant performance improvements on XTREME benchmark tasks.

Effective regularization reduces prediction sensitivity to data augmentations.

Enhances cross-lingual transfer for classification, QA, and sequence labeling.

Abstract

Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation. In addition, we employ model consistency to regularize the models trained with two augmented versions of the same training set. Experimental results on the XTREME benchmark show that our method significantly improves cross-lingual fine-tuning across various tasks, including text classification, question answering, and sequence labeling.