Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

TL;DR

This paper introduces Exact Flow Linear Attention (EFLA), an exact continuous-time formulation that improves stability and performance of linear attention mechanisms in language models.

Contribution

EFLA replaces Euler discretization with an exact flow, preserving structure and complexity, and demonstrates improved robustness and accuracy in experiments.

Findings

EFLA improves robustness under corrupted inputs.

EFLA reduces perplexity in language modeling.

EFLA outperforms SSM and Euler baselines in benchmarks.

Abstract

In this paper, we introduce Exact Flow Linear Attention~(EFLA), an exact-flow formulation of delta-rule linear attention. We show that the delta-rule update can be interpreted as an explicit Euler discretization of an underlying continuous-time system. EFLA replaces this first-order update with the exact closed-form flow. By exploiting the rank-1 structure of the dynamics matrix, both the matrix exponential and the input integral collapse to a simple update that preserves delta-rule linear attention's algebraic structure, parameter count, linear-time complexity, and chunkwise parallelism. This attention mechanism removes the Euler discretization error of the delta-rule dynamics without introducing additional parameters. Experiments on robustness tests, language modeling benchmarks, and the MAD synthetic benchmark show that EFLA improves stability under corrupted and high-energy inputs, reduces perplexity, and achieves stronger downstream performance compared to SSM and Euler-style baselines. These results establish exact-flow integration as a principled and scalable update mechanism for delta-rule linear attention.