Temporal Weights
Adam Kohan, Ed Rietman, Hava Siegelmann

TL;DR
This paper introduces Temporal Weights, a dynamic weight model inspired by biological synapses, integrated with Neural ODEs to improve temporal modeling in neural networks, leading to better performance and efficiency on time series data.
Contribution
It proposes a novel dynamic weight mechanism called Temporal Weights using Neural ODEs, enhancing temporal sequence modeling in neural networks.
Findings
Improved performance on sparse, irregular time series datasets.
Smaller models with higher data efficiency.
Effective modeling of temporal dynamics over sequence length and scale.
Abstract
In artificial neural networks, weights are a static representation of synapses. However, synapses are not static, they have their own interacting dynamics over time. To instill weights with interacting dynamics, we use a model describing synchronization that is capable of capturing core mechanisms of a range of neural and general biological phenomena over time. An ideal fit for these Temporal Weights (TW) are Neural ODEs, with continuous dynamics and a dependency on time. The resulting recurrent neural networks efficiently model temporal dynamics by computing on the ordering of sequences, and the length and scale of time. By adding temporal weights to a model, we demonstrate better performance, smaller models, and data efficiency on sparse, irregularly sampled time series datasets.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Time Series Analysis and Forecasting · Neural dynamics and brain function
