Bridging the Domain Gaps in Context Representations for k-Nearest Neighbor Neural Machine Translation

TL;DR

This paper introduces a method to improve $k$NN-MT by reconstructing key representations in the datastore, aligning them better with downstream domains to enhance translation accuracy.

Contribution

It proposes a novel reviser that refines key representations using semantic and knowledge retention losses, significantly boosting $k$NN-MT performance in domain adaptation.

Findings

Enhanced retrieval accuracy in domain-specific translation tasks.

Improved translation quality demonstrated through extensive experiments.

Effective domain adaptation achieved with the proposed key revision method.

Abstract

$k$-Nearest neighbor machine translation ($k$NN-MT) has attracted increasing attention due to its ability to non-parametrically adapt to new translation domains. By using an upstream NMT model to traverse the downstream training corpus, it is equipped with a datastore containing vectorized key-value pairs, which are retrieved during inference to benefit translation. However, there often exists a significant gap between upstream and downstream domains, which hurts the retrieval accuracy and the final translation quality. To deal with this issue, we propose a novel approach to boost the datastore retrieval of $k$NN-MT by reconstructing the original datastore. Concretely, we design a reviser to revise the key representations, making them better fit for the downstream domain. The reviser is trained using the collected semantically-related key-queries pairs, and optimized by two proposed losses: one is the key-queries semantic distance ensuring each revised key representation is semantically related to its corresponding queries, and the other is an L2-norm loss encouraging revised key representations to effectively retain the knowledge learned by the upstream NMT model. Extensive experiments on domain adaptation tasks demonstrate that our method can effectively boost the datastore retrieval and translation quality of $k$NN-MT.\footnote{Our code is available at \url{https://github.com/DeepLearnXMU/RevisedKey-knn-mt}.}