Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation

TL;DR

This paper demonstrates that fine-tuning only the cross-attention components of pretrained Transformer models is nearly as effective as full fine-tuning for machine translation, enabling efficient transfer learning and zero-shot translation.

Contribution

It reveals that cross-attention fine-tuning suffices for effective transfer learning in machine translation, reducing parameter updates and mitigating catastrophic forgetting.

Findings

Cross-attention fine-tuning nearly matches full model fine-tuning performance.

Limiting fine-tuning to cross-attention yields cross-lingually aligned embeddings.

This approach enables zero-shot translation and reduces parameter storage.

Abstract

We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross-attention when training from scratch. We conduct a series of experiments through fine-tuning a translation model on data where either the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that limiting fine-tuning in this manner yields cross-lingually aligned embeddings. The implications of this finding for researchers and practitioners include a mitigation of catastrophic forgetting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with reduced parameter storage overhead.