Harvesting comparable corpora and mining them for equivalent bilingual sentences using statistical classification and analogy- based heuristics

TL;DR

This paper introduces new methods for extracting parallel bilingual sentences from comparable corpora using statistical classification and analogy-based heuristics, significantly aiding machine translation and cross-lingual retrieval.

Contribution

It presents novel web crawling and analogy-based techniques for mining parallel sentences from comparable corpora, enhancing resource availability for multilingual applications.

Findings

Improved machine translation performance for Polish-English.

Effective web crawling for subject-aligned corpora.

Successful automatic expansion of parallel sentence datasets.

Abstract

Parallel sentences are a relatively scarce but extremely useful resource for many applications including cross-lingual retrieval and statistical machine translation. This research explores our new methodologies for mining such data from previously obtained comparable corpora. The task is highly practical since non-parallel multilingual data exist in far greater quantities than parallel corpora, but parallel sentences are a much more useful resource. Here we propose a web crawling method for building subject-aligned comparable corpora from e.g. Wikipedia dumps and Euronews web page. The improvements in machine translation are shown on Polish-English language pair for various text domains. We also tested another method of building parallel corpora based on comparable corpora data. It lets automatically broad existing corpus of sentences from subject of corpora based on analogies between them.