Automatic Construction of Clean Broad-Coverage Translation Lexicons

TL;DR

This paper introduces an iterative cleaning method for translation lexicons derived from parallel texts, significantly improving their accuracy to over 90% precision and recall by removing indirect association errors.

Contribution

The paper presents a novel iterative lexicon cleaning technique that enhances the quality of translation lexicons by effectively filtering out indirect association errors.

Findings

Achieves over 90% precision and recall in translation lexicons.

Produces dictionaries that are over 99% correct.

Maintains recall while removing most incorrect entries.

Abstract

Word-level translational equivalences can be extracted from parallel texts by surprisingly simple statistical techniques. However, these techniques are easily fooled by {\em indirect associations} --- pairs of unrelated words whose statistical properties resemble those of mutual translations. Indirect associations pollute the resulting translation lexicons, drastically reducing their precision. This paper presents an iterative lexicon cleaning method. On each iteration, most of the remaining incorrect lexicon entries are filtered out, without significant degradation in recall. This lexicon cleaning technique can produce translation lexicons with recall and precision both exceeding 90\%, as well as dictionary-sized translation lexicons that are over 99\% correct.