Additive Interventions Yield Robust Multi-Domain Machine Translation Models

TL;DR

This paper investigates additive interventions for controlling target attributes in neural machine translation across multiple domains, highlighting their robustness to label errors and questioning the benefits of single-domain fine-tuning at larger data scales.

Contribution

It provides a large-scale comparison of additive interventions versus tag-based methods in multi-domain translation, revealing their robustness and challenging previous assumptions about fine-tuning.

Findings

Interventions are robust to label errors.

Performance is similar when domain labels match test domains.

Single-domain fine-tuning's advantage diminishes with more data.

Abstract

Additive interventions are a recently-proposed mechanism for controlling target-side attributes in neural machine translation. In contrast to tag-based approaches which manipulate the raw source sequence, interventions work by directly modulating the encoder representation of all tokens in the sequence. We examine the role of additive interventions in a large-scale multi-domain machine translation setting and compare its performance in various inference scenarios. We find that while the performance difference is small between intervention-based systems and tag-based systems when the domain label matches the test domain, intervention-based systems are robust to label error, making them an attractive choice under label uncertainty. Further, we find that the superiority of single-domain fine-tuning comes under question when training data size is scaled, contradicting previous findings.