Scheduled DropHead: A Regularization Method for Transformer Models

TL;DR

Scheduled DropHead is a structured dropout technique for transformer models that drops entire attention heads during training, improving regularization and efficiency in NLP tasks.

Contribution

It introduces DropHead, a novel structured dropout method for multi-head attention, with an adaptive dropout rate schedule to enhance regularization in transformer models.

Findings

Improves model regularization and reduces overfitting.

Enhances multi-head attention efficiency in NLP tasks.

Proven effective on machine translation and text classification datasets.

Abstract

In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various NLP tasks. In contrast to the conventional dropout mechanisms which randomly drop units or connections, the proposed DropHead is a structured dropout method. It drops entire attention-heads during training and It prevents the multi-head attention model from being dominated by a small portion of attention heads while also reduces the risk of overfitting the training data, thus making use of the multi-head attention mechanism more efficiently. Motivated by recent studies about the learning dynamic of the multi-head attention mechanism, we propose a specific dropout rate schedule to adaptively adjust the dropout rate of DropHead and achieve better regularization effect. Experimental results on both machine translation and text classification benchmark datasets demonstrate the effectiveness of the proposed approach.