Do Transformer Modifications Transfer Across Implementations and Applications?

TL;DR

This paper evaluates numerous Transformer modifications across various NLP tasks, finding that most do not significantly improve performance and highlighting the importance of implementation details.

Contribution

It provides a comprehensive, shared experimental evaluation of Transformer modifications, revealing limited transferability and emphasizing the role of implementation specifics.

Findings

Most modifications do not improve performance significantly

Beneficial variants are often from the same codebase or minor changes

Performance improvements depend heavily on implementation details

Abstract

The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.