BattRAE: Bidimensional Attention-Based Recursive Autoencoders for Learning Bilingual Phrase Embeddings

TL;DR

BattRAE introduces a bidimensional attention-based recursive autoencoder that effectively captures multi-level bilingual phrase interactions, improving translation quality in SMT systems.

Contribution

The paper presents a novel bidimensional attention mechanism integrated with recursive autoencoders for bilingual phrase embedding learning, enhancing semantic similarity measurement.

Findings

Achieves up to 1.63 BLEU point improvement in SMT.

Effectively models multi-level phrase interactions.

Outperforms baseline translation models.

Abstract

In this paper, we propose a bidimensional attention based recursive autoencoder (BattRAE) to integrate clues and sourcetarget interactions at multiple levels of granularity into bilingual phrase representations. We employ recursive autoencoders to generate tree structures of phrases with embeddings at different levels of granularity (e.g., words, sub-phrases and phrases). Over these embeddings on the source and target side, we introduce a bidimensional attention network to learn their interactions encoded in a bidimensional attention matrix, from which we extract two soft attention weight distributions simultaneously. These weight distributions enable BattRAE to generate compositive phrase representations via convolution. Based on the learned phrase representations, we further use a bilinear neural model, trained via a max-margin method, to measure bilingual semantic similarity. To evaluate the effectiveness of BattRAE, we incorporate this semantic similarity as an additional feature into a state-of-the-art SMT system. Extensive experiments on NIST Chinese-English test sets show that our model achieves a substantial improvement of up to 1.63 BLEU points on average over the baseline.