Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation

TL;DR

This paper investigates how different components of neural machine translation models contribute to domain adaptation by freezing parts of the model during continued training, revealing that minimal changes can still yield good performance.

Contribution

It introduces a method of analyzing domain adaptation by freezing subnetworks, showing that minimal component updates can achieve effective adaptation.

Findings

Freezing individual components has little impact on adaptation performance.

Adapting a single component while fixing others yields surprisingly good results.

Continued training results in small model shifts from the out-of-domain baseline.

Abstract

To better understand the effectiveness of continued training, we analyze the major components of a neural machine translation system (the encoder, decoder, and each embedding space) and consider each component's contribution to, and capacity for, domain adaptation. We find that freezing any single component during continued training has minimal impact on performance, and that performance is surprisingly good when a single component is adapted while holding the rest of the model fixed. We also find that continued training does not move the model very far from the out-of-domain model, compared to a sensitivity analysis metric, suggesting that the out-of-domain model can provide a good generic initialization for the new domain.