Physics-inspired Energy Transition Neural Network for Sequence Learning

TL;DR

This paper introduces PETNN, a physics-inspired recurrent neural network that effectively captures long-term dependencies, outperforms transformers on sequence tasks, and offers lower complexity, challenging the dominance of Transformer models.

Contribution

The paper proposes PETNN, a novel physics-inspired recurrent architecture that enhances long-term sequence learning and demonstrates superior performance and efficiency over transformer-based models.

Findings

PETNN outperforms transformer-based methods on various sequence tasks.

PETNN exhibits significantly lower computational complexity.

The study highlights the potential of physics-inspired recurrent models in sequence learning.

Abstract

Recently, the superior performance of Transformers has made them a more robust and scalable solution for sequence modeling than traditional recurrent neural networks (RNNs). However, the effectiveness of Transformer in capturing long-term dependencies is primarily attributed to their comprehensive pair-modeling process rather than inherent inductive biases toward sequence semantics. In this study, we explore the capabilities of pure RNNs and reassess their long-term learning mechanisms. Inspired by the physics energy transition models that track energy changes over time, we propose a effective recurrent structure called the``Physics-inspired Energy Transition Neural Network" (PETNN). We demonstrate that PETNN's memory mechanism effectively stores information over long-term dependencies. Experimental results indicate that PETNN outperforms transformer-based methods across various sequence tasks. Furthermore, owing to its recurrent nature, PETNN exhibits significantly lower complexity. Our study presents an optimal foundational recurrent architecture and highlights the potential for developing effective recurrent neural networks in fields currently dominated by Transformer.