Emergence Transformer: Dynamical Temporal Attention Matters

TL;DR

The paper introduces Emergence Transformer with dynamical temporal attention to modulate coherence in complex systems, demonstrating applications in social dynamics and neural networks.

Contribution

It proposes a novel dynamical temporal attention mechanism that influences emergent coherence in networked systems, extending Transformer capabilities to temporal emergence phenomena.

Findings

Neighbor-DTA promotes oscillatory coherence.

Self-DTA has an optimal attention weight for coherence.

DTA enables emergent continual learning in neural networks.

Abstract

The Transformer, a breakthrough architecture in artificial intelligence, owes its success to the attention mechanism, which utilizes long-range interactions in sequential data, enabling the emergent coherence between large language models (LLMs) and data distributions. However, temporal attention, that is, different forms of long-range interactions in temporal sequences, has rarely been explored in emergence phenomenon of complex systems including oscillatory coherence in quantum, biophysical, or climate systems. Here, by designing dynamical temporal attention (DTA) with time-varying query, key, and value matrices, we propose an Emergence Transformer. This architecture allows each component to interact with its own or its neighbors' past states through dynamical attention kernels, thereby enabling the promotion and/or suppression of the emergent coherence of components. Interestingly, we uncover that neighbor-DTA consistently promotes oscillatory coherence, whereas self-DTA exhibits an optimal attention weight for coherence enhancement, owing to its non-monotonic dependence on network structure. Practically, we demonstrate how DTA reshapes social coherence, suggesting strategies to either enhance agreement or preserve plurality. We further apply DTA to the paradigmatic Hopfield neural network, achieving emergent continual learning without catastrophic forgetting. Together, these results lay a foundation and provide an immediate paradigm for modulating emergence phenomenon in networked dynamics only using DTA.