Rethinking the Value of Transformer Components

TL;DR

This paper evaluates the importance of individual components in Transformer models for translation, revealing key insights and proposing a new training strategy to enhance performance by focusing on important components.

Contribution

It systematically analyzes component contributions in trained Transformers and introduces a novel training method that emphasizes important components for better translation results.

Findings

Certain components are consistently more important across models.

Some components have minimal impact on performance.

A new training strategy improves translation quality by focusing on key components.

Abstract

Transformer becomes the state-of-the-art translation model, while it is not well studied how each intermediate component contributes to the model performance, which poses significant challenges for designing optimal architectures. In this work, we bridge this gap by evaluating the impact of individual component (sub-layer) in trained Transformer models from different perspectives. Experimental results across language pairs, training strategies, and model capacities show that certain components are consistently more important than the others. We also report a number of interesting findings that might help humans better analyze, understand and improve Transformer models. Based on these observations, we further propose a new training strategy that can improves translation performance by distinguishing the unimportant components in training.