NDT: Non-Differential Transformer and Its Application to Sentiment Analysis

TL;DR

This paper introduces the Non-Differential Transformer (NDT), a novel attention mechanism for sentiment analysis that employs positive-only weighted sums of multiple attention maps to improve context understanding.

Contribution

The paper proposes a new transformer architecture that uses a purely additive, positive-weighted attention mechanism inspired by the concept-multiplexing view, contrasting with the noise-canceling approach of differential transformers.

Findings

Achieves competitive sentiment analysis performance on multiple datasets.

Demonstrates the effectiveness of positive-only attention combination.

Provides insights into attention specialization and integration.

Abstract

From customer feedback to social media, understanding human sentiment in text is central to how machines can interact meaningfully with people. However, despite notable progress, accurately capturing sentiment remains a challenging task, which continues to motivate further research in this area. To this end, we introduce Non-Differential Transformer (NDT). It is inspired by (but in contrast to) the state-of-the-art Differential Transformer (DT) model. While standard Transformers can struggle with irrelevant context, the sota DT model uses attention map subtraction, potentially for noise cancellation. We explore an alternative motivation, hypothesizing that benefits may arise from enabling different attention components to specialize on distinct concepts within the text, similar to multiplexing information channels or mixture models, rather than primarily canceling noise via subtraction. Guided by this concept-multiplexing (ConPlex) view, the specific architecture presented in this paper employs a purely additive strategy. It uses only positive weights, learned during training, to ensure constructive combination of these specialized attention perspectives. This design choice explores positive only integration, though our broader framework also shows promise with less constrained linear combinations involving both positive and negative weights. Our model computes attention via this positively weighted sum of multiple distinct attention maps. This allows the model to constructively integrate diverse signals and potentially capture more complex contextual relationships. Competitive performance is achieved by the proposed model for Sentiment Analysis while tested on multiple datasets. We conclude by presenting our results, challenges and future research agenda in this important area of research.