The IIT Bombay English-Hindi Parallel Corpus

TL;DR

The paper introduces the IIT Bombay English-Hindi Parallel Corpus, a large, pre-processed dataset of 1.49 million segments for machine translation, including new data and baseline results, supporting research in Asian language translation.

Contribution

It provides the largest publicly available English-Hindi parallel corpus, combining existing and newly collected data, with baseline translation results and usage in shared tasks.

Findings

Baseline phrase-based SMT and NMT results reported.

Corpus includes 694k new segments not previously available.

Used in shared translation tasks at workshops.

Abstract

We present the IIT Bombay English-Hindi Parallel Corpus. The corpus is a compilation of parallel corpora previously available in the public domain as well as new parallel corpora we collected. The corpus contains 1.49 million parallel segments, of which 694k segments were not previously available in the public domain. The corpus has been pre-processed for machine translation, and we report baseline phrase-based SMT and NMT translation results on this corpus. This corpus has been used in two editions of shared tasks at the Workshop on Asian Language Translation (2016 and 2017). The corpus is freely available for non-commercial research. To the best of our knowledge, this is the largest publicly available English-Hindi parallel corpus.