Language-Independent Representor for Neural Machine Translation

TL;DR

This paper introduces a language-independent representor for neural machine translation that reduces parameters and enhances duality exploration by sharing weights across languages, improving translation quality in various resource settings.

Contribution

Proposes a shared, language-independent representor for NMT that reduces parameters and leverages multi-task training to improve translation performance.

Findings

Significant parameter reduction compared to traditional models.

Improved translation accuracy on resource-rich and low-resource tasks.

Effective exploitation of language duality through joint training.

Abstract

Current Neural Machine Translation (NMT) employs a language-specific encoder to represent the source sentence and adopts a language-specific decoder to generate target translation. This language-dependent design leads to large-scale network parameters and makes the duality of the parallel data underutilized. To address the problem, we propose in this paper a language-independent representor to replace the encoder and decoder by using weight sharing. This shared representor can not only reduce large portion of network parameters, but also facilitate us to fully explore the language duality by jointly training source-to-target, target-to-source, left-to-right and right-to-left translations within a multi-task learning framework. Experiments show that our proposed framework can obtain significant improvements over conventional NMT models on resource-rich and low-resource translation tasks with only a quarter of parameters.