From Fake Focus to Real Precision: Confusion-Driven Adversarial Attention Learning in Transformers

TL;DR

This paper introduces an adversarial attention learning method for Transformers that redistributes attention to task-relevant words, significantly improving sentiment analysis accuracy without manual annotations.

Contribution

The paper proposes a novel adversarial feedback mechanism that dynamically adjusts attention in Transformers, enhancing focus on important words for sentiment analysis.

Findings

Achieves state-of-the-art results on three datasets

Improves large language model performance by 12.6%

Demonstrates effective attention redistribution without manual labels

Abstract

Transformer-based models have been widely adopted for sentiment analysis tasks due to their exceptional ability to capture contextual information. However, these methods often exhibit suboptimal accuracy in certain scenarios. By analyzing their attention distributions, we observe that existing models tend to allocate attention primarily to common words, overlooking less popular yet highly task-relevant terms, which significantly impairs overall performance. To address this issue, we propose an Adversarial Feedback for Attention(AFA) training mechanism that enables the model to automatically redistribute attention weights to appropriate focal points without requiring manual annotations. This mechanism incorporates a dynamic masking strategy that attempts to mask various words to deceive a discriminator, while the discriminator strives to detect significant differences induced by these masks. Additionally, leveraging the sensitivity of Transformer models to token-level perturbations, we employ a policy gradient approach to optimize attention distributions, which facilitates efficient and rapid convergence. Experiments on three public datasets demonstrate that our method achieves state-of-the-art results. Furthermore, applying this training mechanism to enhance attention in large language models yields a further performance improvement of 12.6%