Context-Aware Cross-Attention for Non-Autoregressive Translation

TL;DR

This paper introduces a novel context-aware cross-attention mechanism for non-autoregressive translation models, improving source context capture and translation quality by integrating neighbor source token signals.

Contribution

It proposes a localness perception enhancement in cross-attention to better utilize source context in NAT models, addressing a key limitation.

Findings

Consistent improvement over strong NAT baselines.

Better exploitation of source contexts with local and global info.

Enhanced translation quality demonstrated on multiple datasets.

Abstract

Non-autoregressive translation (NAT) significantly accelerates the inference process by predicting the entire target sequence. However, due to the lack of target dependency modelling in the decoder, the conditional generation process heavily depends on the cross-attention. In this paper, we reveal a localness perception problem in NAT cross-attention, for which it is difficult to adequately capture source context. To alleviate this problem, we propose to enhance signals of neighbour source tokens into conventional cross-attention. Experimental results on several representative datasets show that our approach can consistently improve translation quality over strong NAT baselines. Extensive analyses demonstrate that the enhanced cross-attention achieves better exploitation of source contexts by leveraging both local and global information.