Can Latent Alignments Improve Autoregressive Machine Translation?

TL;DR

This paper investigates whether latent alignment objectives can enhance autoregressive machine translation models, finding that they often lead to degenerate models due to incompatibility with teacher forcing, supported by theoretical analysis.

Contribution

The paper demonstrates that latent alignment objectives are incompatible with teacher forcing in autoregressive models and provides a theoretical explanation for the observed degenerate solutions.

Findings

Latent alignment objectives cause degenerate autoregressive models.

Theoretical proof shows incompatibility with teacher forcing.

Empirical results confirm the limitations of latent alignments in this setting.

Abstract

Latent alignment objectives such as CTC and AXE significantly improve non-autoregressive machine translation models. Can they improve autoregressive models as well? We explore the possibility of training autoregressive machine translation models with latent alignment objectives, and observe that, in practice, this approach results in degenerate models. We provide a theoretical explanation for these empirical results, and prove that latent alignment objectives are incompatible with teacher forcing.