Multi-Level Contrastive Learning for Cross-Lingual Alignment

TL;DR

This paper introduces a multi-level contrastive learning framework that enhances cross-lingual transfer in pre-trained models by integrating word-level information and a novel CZ-NCE loss, leading to improved performance on zero-shot tasks.

Contribution

It presents a novel multi-level contrastive learning approach with word-level alignment and CZ-NCE loss to boost cross-lingual capabilities of mBERT.

Findings

Significant improvement in cross-lingual transfer performance.

Outperforms previous models on Xtreme benchmark tasks.

Effective use of word-level information in contrastive learning.

Abstract

Cross-language pre-trained models such as multilingual BERT (mBERT) have achieved significant performance in various cross-lingual downstream NLP tasks. This paper proposes a multi-level contrastive learning (ML-CTL) framework to further improve the cross-lingual ability of pre-trained models. The proposed method uses translated parallel data to encourage the model to generate similar semantic embeddings for different languages. However, unlike the sentence-level alignment used in most previous studies, in this paper, we explicitly integrate the word-level information of each pair of parallel sentences into contrastive learning. Moreover, cross-zero noise contrastive estimation (CZ-NCE) loss is proposed to alleviate the impact of the floating-point error in the training process with a small batch size. The proposed method significantly improves the cross-lingual transfer ability of our basic model (mBERT) and outperforms on multiple zero-shot cross-lingual downstream tasks compared to the same-size models in the Xtreme benchmark.