Low-resource Neural Machine Translation with Cross-modal Alignment

TL;DR

This paper introduces a cross-modal contrastive learning approach to improve low-resource neural machine translation by leveraging visual modality, enabling better cross-lingual alignment with minimal image-text pairs.

Contribution

It proposes a novel cross-modal contrastive learning method that aligns multiple low-resource languages with a high-resource language using visual data, enhancing translation performance.

Findings

Effective cross-modal and cross-lingual alignment achieved

Significant improvements over text-only baseline in zero-shot scenarios

Works well with limited image-text pairs

Abstract

How to achieve neural machine translation with limited parallel data? Existing techniques often rely on large-scale monolingual corpora, which is impractical for some low-resource languages. In this paper, we turn to connect several low-resource languages to a particular high-resource one by additional visual modality. Specifically, we propose a cross-modal contrastive learning method to learn a shared space for all languages, where both a coarse-grained sentence-level objective and a fine-grained token-level one are introduced. Experimental results and further analysis show that our method can effectively learn the cross-modal and cross-lingual alignment with a small amount of image-text pairs and achieves significant improvements over the text-only baseline under both zero-shot and few-shot scenarios.