Incremental Adaptation Strategies for Neural Network Language Models

TL;DR

This paper introduces efficient incremental adaptation techniques for neural network language models, enabling rapid updates with small data sets without overfitting, thus improving translation quality.

Contribution

It proposes two novel methods—continued training on resampled data and insertion of adaptation layers—for fast neural model adaptation.

Findings

Both methods are computationally efficient and fast.

They significantly improve translation quality.

They prevent overfitting on small adaptation datasets.

Abstract

It is today acknowledged that neural network language models outperform backoff language models in applications like speech recognition or statistical machine translation. However, training these models on large amounts of data can take several days. We present efficient techniques to adapt a neural network language model to new data. Instead of training a completely new model or relying on mixture approaches, we propose two new methods: continued training on resampled data or insertion of adaptation layers. We present experimental results in an CAT environment where the post-edits of professional translators are used to improve an SMT system. Both methods are very fast and achieve significant improvements without overfitting the small adaptation data.