Automatic Discovery of Non-Compositional Compounds in Parallel Data

TL;DR

This paper introduces an efficient automatic method to identify non-compositional word sequences in parallel data, enhancing machine translation by capturing units translated as a whole, applicable across various data types.

Contribution

The paper presents a novel, data-agnostic approach for discovering non-compositional compounds in parallel data using statistical translation models, improving MT quality.

Findings

Discovered hundreds of non-compositional compounds per iteration

Constructed longer compounds from shorter ones

Improved machine translation output quality

Abstract

Automatic segmentation of text into minimal content-bearing units is an unsolved problem even for languages like English. Spaces between words offer an easy first approximation, but this approximation is not good enough for machine translation (MT), where many word sequences are not translated word-for-word. This paper presents an efficient automatic method for discovering sequences of words that are translated as a unit. The method proceeds by comparing pairs of statistical translation models induced from parallel texts in two languages. It can discover hundreds of non-compositional compounds on each iteration, and constructs longer compounds out of shorter ones. Objective evaluation on a simple machine translation task has shown the method's potential to improve the quality of MT output. The method makes few assumptions about the data, so it can be applied to parallel data other than parallel texts, such as word spellings and pronunciations.