Adversarial Self-Attention for Language Understanding

TL;DR

This paper introduces Adversarial Self-Attention (ASA), a novel mechanism that enhances Transformer models by reducing reliance on spurious features, thereby improving their robustness and generalization across various language understanding tasks.

Contribution

It proposes ASA, an adversarially biased self-attention mechanism that suppresses reliance on specific features and promotes broader semantic exploration in Transformer models.

Findings

ASA improves pre-training performance over naive training.

ASA-empowered models outperform naive models in generalization.

Enhanced robustness and robustness in language understanding tasks.

Abstract

Deep neural models (e.g. Transformer) naturally learn spurious features, which create a ``shortcut'' between the labels and inputs, thus impairing the generalization and robustness. This paper advances the self-attention mechanism to its robust variant for Transformer-based pre-trained language models (e.g. BERT). We propose \textit{Adversarial Self-Attention} mechanism (ASA), which adversarially biases the attentions to effectively suppress the model reliance on features (e.g. specific keywords) and encourage its exploration of broader semantics. We conduct a comprehensive evaluation across a wide range of tasks for both pre-training and fine-tuning stages. For pre-training, ASA unfolds remarkable performance gains compared to naive training for longer steps. For fine-tuning, ASA-empowered models outweigh naive models by a large margin considering both generalization and robustness.