Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond

TL;DR

This paper introduces a novel warm-up based framework that combines zero-shot and translation-based methods to improve cross-lingual sequence tagging performance using multilingual pre-trained models.

Contribution

It proposes a tailored warm-up mechanism and a refined adaptation approach that enhances cross-lingual transfer for sequence tagging tasks.

Findings

Improved performance on nine target languages.

Effective combination of zero-shot and translation-based approaches.

Beneficial for various sequence tagging tasks.

Abstract

Cross-lingual adaptation with multilingual pre-trained language models (mPTLMs) mainly consists of two lines of works: zero-shot approach and translation-based approach, which have been studied extensively on the sequence-level tasks. We further verify the efficacy of these cross-lingual adaptation approaches by evaluating their performances on more fine-grained sequence tagging tasks. After re-examining their strengths and drawbacks, we propose a novel framework to consolidate the zero-shot approach and the translation-based approach for better adaptation performance. Instead of simply augmenting the source data with the machine-translated data, we tailor-make a warm-up mechanism to quickly update the mPTLMs with the gradients estimated on a few translated data. Then, the adaptation approach is applied to the refined parameters and the cross-lingual transfer is performed in a warm-start way. The experimental results on nine target languages demonstrate that our method is beneficial to the cross-lingual adaptation of various sequence tagging tasks.