Folded Context Condensation in Path Integral Formalism for Infinite Context Transformers

TL;DR

This paper introduces a novel Transformer formulation based on Path Integral formalism, leading to more efficient long-term information retention and linear memory scaling, validated through retrieval and summarization tasks.

Contribution

It reinterprets Transformer components within Path Integral formalism, resulting in a more compact, memory-efficient architecture for long sequences.

Findings

Memory usage scales linearly with sequence length

Preserves historical information effectively

Outperforms standard attention in long-term retention

Abstract

In this work, we present a generalized formulation of the Transformer algorithm by reinterpreting its core mechanisms within the framework of Path Integral formalism. In this perspective, the attention mechanism is recast as a process that integrates all possible transition paths leading to future token states, with temporal evolution governed by the Feed-Forward Network. By systematically mapping each component of the Transformer to its counterpart in the Path Integral formulation, we obtain a more compact and efficient representation, in which the contextual information of a sequence is condensed into memory-like segments. These segments are recurrently processed across Transformer layers, enabling more effective long-term information retention. We validate the effectiveness of this approach through the Passkey retrieval task and a summarization task, demonstrating that the proposed method preserves historical information while exhibiting memory usage that scales linearly with sequence length. This contrasts with the non-linear memory growth typically observed in standard attention mechanisms. We expect that this quantum-inspired generalization of the Transformer architecture will open new avenues for enhancing both the efficiency and expressiveness of future Transformer models.