Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation

TL;DR

This paper explores methods for encoding sentence positions in context-aware neural machine translation, demonstrating benefits in English-Russian translation with specific encoding and training strategies, but not in English-German.

Contribution

It introduces and compares various sentence position encoding methods within Transformer models for context-aware translation, highlighting their effectiveness under certain training conditions.

Findings

Sentence position encoding improves English-Russian translation quality.

Benefits depend on training with a context-discounted loss.

No significant improvement observed in English-German translation.

Abstract

Context-aware translation can be achieved by processing a concatenation of consecutive sentences with the standard Transformer architecture. This paper investigates the intuitive idea of providing the model with explicit information about the position of the sentences contained in the concatenation window. We compare various methods to encode sentence positions into token representations, including novel methods. Our results show that the Transformer benefits from certain sentence position encoding methods on English to Russian translation if trained with a context-discounted loss (Lupo et al., 2022). However, the same benefits are not observed in English to German. Further empirical efforts are necessary to define the conditions under which the proposed approach is beneficial.