How Effective are State Space Models for Machine Translation?

TL;DR

This paper rigorously compares state space models and transformers for machine translation, showing that certain linear recurrent models like Mamba are highly competitive, especially with longer sequences and when combined with attention mechanisms.

Contribution

It provides the first comprehensive experimental comparison between transformers and linear recurrent models for MT, highlighting the strengths of Mamba and hybrid models.

Findings

Mamba is highly competitive with transformers on sentence and paragraph datasets.

Incorporating attention into Mamba improves translation quality and robustness.

Both models benefit from training on longer sequences.

Abstract

Transformers are the current architecture of choice for NLP, but their attention layers do not scale well to long contexts. Recent works propose to replace attention with linear recurrent layers -- this is the case for state space models, which enjoy efficient training and inference. However, it remains unclear whether these models are competitive with transformers in machine translation (MT). In this paper, we provide a rigorous and comprehensive experimental comparison between transformers and linear recurrent models for MT. Concretely, we experiment with RetNet, Mamba, and hybrid versions of Mamba which incorporate attention mechanisms. Our findings demonstrate that Mamba is highly competitive with transformers on sentence and paragraph-level datasets, where in the latter both models benefit from shifting the training distribution towards longer sequences. Further analysis show that integrating attention into Mamba improves translation quality, robustness to sequence length extrapolation, and the ability to recall named entities.