# Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

## TL;DR

This paper presents a unified multilingual sentence embedding system trained on 93 languages, enabling zero-shot transfer for various NLP tasks and demonstrating strong performance across multiple benchmarks and a new multilingual test set.

## Contribution

The authors introduce a single BiLSTM-based architecture with shared vocabulary for 93 languages, facilitating zero-shot transfer and multilingual similarity search.

## Key findings

- Effective cross-lingual transfer in NLI, classification, and mining tasks
- Strong results on XNLI, MLDoc, and BUCC datasets
- High-quality embeddings for low-resource languages

## Abstract

We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared BPE vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI dataset), cross-lingual document classification (MLDoc dataset) and parallel corpus mining (BUCC dataset) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low-resource languages. Our implementation, the pre-trained encoder and the multilingual test set are available at https://github.com/facebookresearch/LASER