A Multilingual Parallel Corpora Collection Effort for Indian Languages

TL;DR

This paper introduces a large, multilingual parallel corpus for ten Indian languages, enhancing resources for low-resource languages and facilitating machine translation and cross-lingual tasks.

Contribution

It presents a significant extension of existing corpora for Indian languages, including a test set and methods using neural networks for corpus construction.

Findings

Extended corpora covering 10 Indian languages.

A new test corpus for validation.

Use of neural network methods for corpus creation.

Abstract

We present sentence aligned parallel corpora across 10 Indian Languages - Hindi, Telugu, Tamil, Malayalam, Gujarati, Urdu, Bengali, Oriya, Marathi, Punjabi, and English - many of which are categorized as low resource. The corpora are compiled from online sources which have content shared across languages. The corpora presented significantly extends present resources that are either not large enough or are restricted to a specific domain (such as health). We also provide a separate test corpus compiled from an independent online source that can be independently used for validating the performance in 10 Indian languages. Alongside, we report on the methods of constructing such corpora using tools enabled by recent advances in machine translation and cross-lingual retrieval using deep neural network based methods.