Consequences of Slow Neural Dynamics for Incremental Learning

TL;DR

This paper shows that neural networks with slow, recurrent dynamics learn categories more efficiently from temporally smooth data and can separate fast and slow environmental signals, highlighting the role of cortical temporal autocorrelation.

Contribution

It demonstrates that slow neural dynamics and recurrence serve as an inductive bias, improving learning efficiency and representation separation in temporally correlated data.

Findings

Slow recurrent networks outperform feedforward models in categorization tasks.

Networks with recurrence and gating un-mix fast and slow data sources.

Temporal autocorrelation in neural dynamics enhances learning and representation.

Abstract

In the human brain, internal states are often correlated over time (due to local recurrence and other intrinsic circuit properties), punctuated by abrupt transitions. At first glance, temporal smoothness of internal states presents a problem for learning input-output mappings (e.g. category labels for images), because the internal representation of the input will contain a mixture of current input and prior inputs. However, when training with naturalistic data (e.g. movies) there is also temporal autocorrelation in the input. How does the temporal "smoothness" of internal states affect the efficiency of learning when the training data are also temporally smooth? How does it affect the kinds of representations that are learned? We found that, when trained with temporally smooth data, "slow" neural networks (equipped with linear recurrence and gating mechanisms) learned to categorize more efficiently than feedforward networks. Furthermore, networks with linear recurrence and multi-timescale gating could learn internal representations that "un-mixed" quickly-varying and slowly-varying data sources. Together, these findings demonstrate how a fundamental property of cortical dynamics (their temporal autocorrelation) can serve as an inductive bias, leading to more efficient category learning and to the representational separation of fast and slow sources in the environment.