Manual Annotation of Translational Equivalence: The Blinker Project

TL;DR

This paper presents a manually annotated dataset linking approximately sixteen thousand words between French and English Bible texts, facilitating research in translation, lexical semantics, and word-sense disambiguation.

Contribution

It introduces a new high-quality bilingual annotation dataset, along with a specialized tool and methodology to ensure consistency and reliability in manual translation equivalence annotation.

Findings

Annotations are reasonably reliable with high inter-annotator agreement

The annotation process is replicable and scalable

The dataset supports multiple research applications in translation and semantics

Abstract

Bilingual annotators were paid to link roughly sixteen thousand corresponding words between on-line versions of the Bible in modern French and modern English. These annotations are freely available to the research community from http://www.cis.upenn.edu/~melamed . The annotations can be used for several purposes. First, they can be used as a standard data set for developing and testing translation lexicons and statistical translation models. Second, researchers in lexical semantics will be able to mine the annotations for insights about cross-linguistic lexicalization patterns. Third, the annotations can be used in research into certain recently proposed methods for monolingual word-sense disambiguation. This paper describes the annotated texts, the specially-designed annotation tool, and the strategies employed to increase the consistency of the annotations. The annotation process was repeated five times by different annotators. Inter-annotator agreement rates indicate that the annotations are reasonably reliable and that the method is easy to replicate.