XNLI: Evaluating Cross-lingual Sentence Representations

TL;DR

This paper introduces XNLI, a multilingual dataset for evaluating cross-lingual sentence understanding, extending MultiNLI to 15 languages, and provides baseline models demonstrating the challenge of cross-lingual transfer.

Contribution

The creation of the XNLI dataset for cross-lingual evaluation and baseline models for multilingual sentence understanding.

Findings

Translating test data yields the best baseline performance.

XNLI is a practical, challenging evaluation suite.

Multilingual models still lag behind monolingual performance.

Abstract

State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in cross-lingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.