DINT Transformer
Yueyang Cang, Yuhang Liu, Xiaoteng Zhang, Erlu Zhao, Li Shi

TL;DR
DINT Transformer enhances local attention with global context modeling and improved numerical stability, leading to better performance in long-context language tasks.
Contribution
It introduces a differential-integral mechanism to incorporate global importance scores and enforces row-normalized attention matrices for stability.
Findings
Outperforms previous models in long-context language modeling
Demonstrates improved robustness in key information retrieval
Achieves higher accuracy across multiple practical applications
Abstract
DIFF Transformer addresses the issue of irrelevant context interference by introducing a differential attention mechanism that enhances the robustness of local attention. However, it has two critical limitations: the lack of global context modeling, which is essential for identifying globally significant tokens, and numerical instability due to the absence of strict row normalization in the attention matrix. To overcome these challenges, we propose DINT Transformer, which extends DIFF Transformer by incorporating a differential-integral mechanism. By computing global importance scores and integrating them into the attention matrix, DINT Transformer improves its ability to capture global dependencies. Moreover, the unified parameter design enforces row-normalized attention matrices, improving numerical stability. Experimental results demonstrate that DINT Transformer excels in accuracy…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsSensor Technology and Measurement Systems · Physics and Engineering Research Articles
MethodsAttention Is All You Need · Softmax · Adam · Residual Connection · Dropout · Absolute Position Encodings · Byte Pair Encoding · Linear Layer · Multi-Head Attention · Position-Wise Feed-Forward Layer
