Bilingual Document Alignment with Latent Semantic Indexing

TL;DR

This paper presents a bilingual document alignment method using cross-lingual Latent Semantic Indexing, achieving high recall rates in aligning English and French web pages without extensive in-domain data.

Contribution

The paper introduces a novel bilingual alignment approach leveraging latent semantic indexing and cosine similarity, improving alignment accuracy without relying heavily on in-domain data.

Findings

Achieves approximately 88% recall without in-domain data.

Improves to 93% recall when in-domain data is used.

Proposes a new evaluation method accounting for duplicates.

Abstract

We apply cross-lingual Latent Semantic Indexing to the Bilingual Document Alignment Task at WMT16. Reduced-rank singular value decomposition of a bilingual term-document matrix derived from known English/French page pairs in the training data allows us to map monolingual documents into a joint semantic space. Two variants of cosine similarity between the vectors that place each document into the joint semantic space are combined with a measure of string similarity between corresponding URLs to produce 1:1 alignments of English/French web pages in a variety of domains. The system achieves a recall of ca. 88% if no in-domain data is used for building the latent semantic model, and 93% if such data is included. Analysing the system's errors on the training data, we argue that evaluating aligner performance based on exact URL matches under-estimates their true performance and propose an alternative that is able to account for duplicates and near-duplicates in the underlying data.