Modeling Coherence for Neural Machine Translation with Dynamic and Topic Caches

TL;DR

This paper introduces a cache-based neural machine translation model that captures cross-sentence and topical context to improve coherence, combining cache-derived probabilities with NMT predictions for better translation quality.

Contribution

It proposes a novel cache-based neural model with dynamic and topic caches, trained end-to-end, to enhance coherence in NMT systems, outperforming existing baselines.

Findings

Significant improvements over state-of-the-art NMT baselines.

Effective modeling of cross-sentence and topical context.

Enhanced translation coherence and quality.

Abstract

Sentences in a well-formed text are connected to each other via various links to form the cohesive structure of the text. Current neural machine translation (NMT) systems translate a text in a conventional sentence-by-sentence fashion, ignoring such cross-sentence links and dependencies. This may lead to generate an incoherent target text for a coherent source text. In order to handle this issue, we propose a cache-based approach to modeling coherence for neural machine translation by capturing contextual information either from recently translated sentences or the entire document. Particularly, we explore two types of caches: a dynamic cache, which stores words from the best translation hypotheses of preceding sentences, and a topic cache, which maintains a set of target-side topical words that are semantically related to the document to be translated. On this basis, we build a new layer to score target words in these two caches with a cache-based neural model. Here the estimated probabilities from the cache-based neural model are combined with NMT probabilities into the final word prediction probabilities via a gating mechanism. Finally, the proposed cache-based neural model is trained jointly with NMT system in an end-to-end manner. Experiments and analysis presented in this paper demonstrate that the proposed cache-based model achieves substantial improvements over several state-of-the-art SMT and NMT baselines.