Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast

TL;DR

This paper introduces a novel dual momentum contrast method to align cross-lingual sentence embeddings, achieving state-of-the-art results in multilingual similarity and mining tasks by improving negative sample quality.

Contribution

It adapts MoCo contrastive learning to cross-lingual sentence embedding alignment, enhancing negative sample quality over previous batch-based methods.

Findings

Achieves new state-of-the-art on Tatoeba en-zh similarity search

Improves performance on BUCC en-zh bitext mining

Enhances semantic textual similarity across 7 datasets

Abstract

In this paper, we propose to align sentence representations from different languages into a unified embedding space, where semantic similarities (both cross-lingual and monolingual) can be computed with a simple dot product. Pre-trained language models are fine-tuned with the translation ranking task. Existing work (Feng et al., 2020) uses sentences within the same batch as negatives, which can suffer from the issue of easy negatives. We adapt MoCo (He et al., 2020) to further improve the quality of alignment. As the experimental results show, the sentence representations produced by our model achieve the new state-of-the-art on several tasks, including Tatoeba en-zh similarity search (Artetxe and Schwenk, 2019b), BUCC en-zh bitext mining, and semantic textual similarity on 7 datasets.