Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers

TL;DR

This paper introduces Continuous_Time Attention, a PDE-guided extension to Transformer attention mechanisms, enabling better modeling of long sequences through dynamic, smooth attention weights that improve dependency capture and training stability.

Contribution

It presents a novel PDE-based framework for attention in Transformers, allowing weights to evolve over pseudo_time, which improves long-range dependency modeling and training stability.

Findings

Consistent performance improvements over standard Transformers on long sequence tasks.

Theoretical analysis shows PDE-based attention offers better optimization landscapes.

Enhanced ability to model global coherence in long sequences.

Abstract

We propose a novel framework, Continuous_Time Attention, which infuses partial differential equations (PDEs) into the Transformer's attention mechanism to address the challenges of extremely long input sequences. Instead of relying solely on a static attention matrix, we allow attention weights to evolve over a pseudo_time dimension via diffusion, wave, or reaction_diffusion dynamics. This mechanism systematically smooths local noise, enhances long_range dependencies, and stabilizes gradient flow. Theoretically, our analysis shows that PDE_based attention leads to better optimization landscapes and polynomial rather than exponential decay of distant interactions. Empirically, we benchmark our method on diverse experiments_demonstrating consistent gains over both standard and specialized long sequence Transformer variants. Our findings highlight the potential of PDE_based formulations to enrich attention mechanisms with continuous_time dynamics and global coherence.