Realistic Zero-Shot Cross-Lingual Transfer in Legal Topic Classification

TL;DR

This paper introduces a new non-parallel dataset for legal topic classification and demonstrates that translation-based methods and a bilingual teacher-student approach significantly improve zero-shot cross-lingual transfer performance.

Contribution

It creates a non-parallel version of the MultiEURLEX dataset and proposes a bilingual teacher-student method that outperforms previous zero-shot transfer techniques.

Findings

Translation-based methods outperform cross-lingual fine-tuning.

Bilingual teacher-student approach improves zero-shot transfer.

New dataset version enables more realistic evaluation.

Abstract

We consider zero-shot cross-lingual transfer in legal topic classification using the recent MultiEURLEX dataset. Since the original dataset contains parallel documents, which is unrealistic for zero-shot cross-lingual transfer, we develop a new version of the dataset without parallel documents. We use it to show that translation-based methods vastly outperform cross-lingual fine-tuning of multilingually pre-trained models, the best previous zero-shot transfer method for MultiEURLEX. We also develop a bilingual teacher-student zero-shot transfer approach, which exploits additional unlabeled documents of the target language and performs better than a model fine-tuned directly on labeled target language documents.