DINT Transformer

TL;DR

DINT Transformer enhances local attention with global context modeling and improved numerical stability, leading to better performance in long-context language tasks.

Contribution

It introduces a differential-integral mechanism to incorporate global importance scores and enforces row-normalized attention matrices for stability.

Findings

Outperforms previous models in long-context language modeling

Demonstrates improved robustness in key information retrieval

Achieves higher accuracy across multiple practical applications

Abstract

DIFF Transformer addresses the issue of irrelevant context interference by introducing a differential attention mechanism that enhances the robustness of local attention. However, it has two critical limitations: the lack of global context modeling, which is essential for identifying globally significant tokens, and numerical instability due to the absence of strict row normalization in the attention matrix. To overcome these challenges, we propose DINT Transformer, which extends DIFF Transformer by incorporating a differential-integral mechanism. By computing global importance scores and integrating them into the attention matrix, DINT Transformer improves its ability to capture global dependencies. Moreover, the unified parameter design enforces row-normalized attention matrices, improving numerical stability. Experimental results demonstrate that DINT Transformer excels in accuracy and robustness across various practical applications, such as long-context language modeling and key information retrieval. These results position DINT Transformer as a highly effective and promising architecture.