XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation

TL;DR

XGLUE is a comprehensive multilingual benchmark dataset designed to evaluate cross-lingual models across understanding and generation tasks, offering diverse tasks and multi-language labels to advance multilingual NLP research.

Contribution

The paper introduces XGLUE, a new benchmark dataset with 11 diverse tasks and multi-language labels, and extends a cross-lingual model to evaluate performance across understanding and generation.

Findings

Unicoder achieves strong baseline performance on XGLUE.

Multilingual BERT, XLM, and XLM-R are evaluated for comparison.

XGLUE covers both understanding and generation tasks across multiple languages.

Abstract

In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in English for natural language understanding tasks only, XGLUE has two main advantages: (1) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (2) for each task, it provides labeled data in multiple languages. We extend a recent cross-lingual pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison.