# Unbabel's Submission to the WMT2019 APE Shared Task: BERT-based Encoder-Decoder for Automatic Post-Editing

## TL;DR

This paper presents a BERT-based encoder-decoder model for automatic post-editing of machine translation, achieving state-of-the-art results in English-German APE by leveraging pre-trained language models and a conservativeness constraint.

## Contribution

The paper introduces a novel BERT-based encoder-decoder architecture with a conservativeness factor for APE, improving translation quality and setting new shared task benchmarks.

## Key findings

- Significant TER and BLEU improvements over baseline systems.
- Achieved state-of-the-art results in English-German APE.
- Effective use of pre-trained BERT models for post-editing.

## Abstract

This paper describes Unbabel's submission to the WMT2019 APE Shared Task for the English-German language pair. Following the recent rise of large, powerful, pre-trained models, we adapt the BERT pretrained model to perform Automatic Post-Editing in an encoder-decoder framework. Analogously to dual-encoder architectures we develop a BERT-based encoder-decoder (BED) model in which a single pretrained BERT encoder receives both the source src and machine translation tgt strings. Furthermore, we explore a conservativeness factor to constrain the APE system to perform fewer edits. As the official results show, when trained on a weighted combination of in-domain and artificial training data, our BED system with the conservativeness penalty improves significantly the translations of a strong Neural Machine Translation system by $-0.78$ and $+1.23$ in terms of TER and BLEU, respectively. Finally, our submission achieves a new state-of-the-art, ex-aequo, in English-German APE of NMT.