Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings

TL;DR

This paper compares data transfer and model transfer techniques for zero-resource cross-lingual sequence labelling, finding that model transfer with multilingual models generally outperforms data transfer, especially when high-capacity models are available.

Contribution

It provides an in-depth experimental comparison of data-based and model-based cross-lingual transfer methods, highlighting the superiority of model transfer with multilingual models in zero-shot settings.

Findings

Model transfer with multilingual models outperforms data transfer.

Translation-based data transfer can introduce discrepancies affecting performance.

Data transfer remains viable without high-capacity multilingual models.

Abstract

Zero-resource cross-lingual transfer approaches aim to apply supervised models from a source language to unlabelled target languages. In this paper we perform an in-depth study of the two main techniques employed so far for cross-lingual zero-resource sequence labelling, based either on data or model transfer. Although previous research has proposed translation and annotation projection (data-based cross-lingual transfer) as an effective technique for cross-lingual sequence labelling, in this paper we experimentally demonstrate that high capacity multilingual language models applied in a zero-shot (model-based cross-lingual transfer) setting consistently outperform data-based cross-lingual transfer approaches. A detailed analysis of our results suggests that this might be due to important differences in language use. More specifically, machine translation often generates a textual signal which is different to what the models are exposed to when using gold standard data, which affects both the fine-tuning and evaluation processes. Our results also indicate that data-based cross-lingual transfer approaches remain a competitive option when high-capacity multilingual language models are not available.