IndicXNLI: Evaluating Multilingual Inference for Indian Languages

TL;DR

This paper introduces IndicXNLI, a new benchmark dataset for natural language inference in 11 Indian languages, created through high-quality machine translation of the English XNLI dataset, to evaluate multilingual models.

Contribution

The paper presents IndicXNLI, a novel NLI dataset for Indian languages, and analyzes cross-lingual transfer techniques using various pre-trained language models.

Findings

Insights into model performance across languages

Impact of multi-linguality and input mixing

Evaluation of transfer techniques

Abstract

While Indic NLP has made rapid advances recently in terms of the availability of corpora and pre-trained models, benchmark datasets on standard NLU tasks are limited. To this end, we introduce IndicXNLI, an NLI dataset for 11 Indic languages. It has been created by high-quality machine translation of the original English XNLI dataset and our analysis attests to the quality of IndicXNLI. By finetuning different pre-trained LMs on this IndicXNLI, we analyze various cross-lingual transfer techniques with respect to the impact of the choice of language models, languages, multi-linguality, mix-language input, etc. These experiments provide us with useful insights into the behaviour of pre-trained models for a diverse set of languages.