FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models

TL;DR

FreeTransfer-X introduces a label-free cross-lingual transfer method using off-the-shelf models and knowledge distillation, enabling effective multilingual knowledge transfer without requiring labeled data or sensitive labels.

Contribution

It proposes a novel CLT framework, FreeTransfer-X, that transfers knowledge from off-the-shelf models using a two-step knowledge distillation process, bypassing the need for labeled data.

Findings

Outperforms neural machine translation baselines

Reduces annotation costs and preserves privacy

Compatible with various network architectures

Abstract

Cross-lingual transfer (CLT) is of various applications. However, labeled cross-lingual corpus is expensive or even inaccessible, especially in the fields where labels are private, such as diagnostic results of symptoms in medicine and user profiles in business. Nevertheless, there are off-the-shelf models in these sensitive fields. Instead of pursuing the original labels, a workaround for CLT is to transfer knowledge from the off-the-shelf models without labels. To this end, we define a novel CLT problem named FreeTransfer-X that aims to achieve knowledge transfer from the off-the-shelf models in rich-resource languages. To address the problem, we propose a 2-step knowledge distillation (KD, Hinton et al., 2015) framework based on multilingual pre-trained language models (mPLM). The significant improvement over strong neural machine translation (NMT) baselines demonstrates the effectiveness of the proposed method. In addition to reducing annotation cost and protecting private labels, the proposed method is compatible with different networks and easy to be deployed. Finally, a range of analyses indicate the great potential of the proposed method.