DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales

TL;DR

DeepSITH introduces a biologically-inspired neural network architecture that effectively captures temporal relationships across multiple time scales, outperforming traditional RNNs like LSTMs on various time series tasks.

Contribution

The paper presents DeepSITH, a novel neural network with SITH modules that respond over geometrically-spaced time constants, enabling efficient learning across diverse time scales.

Findings

DeepSITH outperforms LSTMs on several time series prediction tasks.

SITH modules effectively encode information over multiple time scales.

DeepSITH achieves state-of-the-art results in temporal learning benchmarks.

Abstract

Extracting temporal relationships over a range of scales is a hallmark of human perception and cognition -- and thus it is a critical feature of machine learning applied to real-world problems. Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs). This paper introduces DeepSITH, a network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. SITH modules respond to their inputs with a geometrically-spaced set of time constants, enabling the DeepSITH network to learn problems along a continuum of time-scales. We compare DeepSITH to LSTMs and other recent RNNs on several time series prediction and decoding tasks. DeepSITH achieves state-of-the-art performance on these problems.