Neutron: An Implementation of the Transformer Translation Model and its Variants

TL;DR

This paper presents Neutron, a highly optimized and modular implementation of the Transformer translation model and its variants, facilitating research and industrial applications with improved performance and readability.

Contribution

It introduces Neutron, an implementation that is easy to modify, optimized, and includes recent Transformer variants, enhancing usability and research flexibility.

Findings

Comparable performance to existing Transformer models

Easy to modify and extend with recent variants

Optimized for better parallelization and readability

Abstract

The Transformer translation model is easier to parallelize and provides better performance compared to recurrent seq2seq models, which makes it popular among industry and research community. We implement the Neutron in this work, including the Transformer model and its several variants from most recent researches. It is highly optimized, easy to modify and provides comparable performance with interesting features while keeping readability.