On the dynamics of convolutional recurrent neural networks near their critical point

TL;DR

This paper analyzes the dynamic behavior of a convolutional recurrent neural network near its critical point, revealing how input signals influence activity, relaxation timescales, and propagation lengths through analytical solutions.

Contribution

It provides analytical insights into the steady states and signal propagation dynamics of convolutional recurrent networks at criticality, especially with unitary kernels and sigmoidal activations.

Findings

Inputs generate ongoing activity controlling signal propagation.

Relaxation timescales and length-scales diverge as input approaches zero.

Analytical solutions describe steady states under oscillatory forcing.

Abstract

We examine the dynamical properties of a single-layer convolutional recurrent network with a smooth sigmoidal activation function, for small values of the inputs and when the convolution kernel is unitary, so all eigenvalues lie exactly at the unit circle. Such networks have a variety of hallmark properties: the outputs depend on the inputs via compressive nonlinearities such as cubic roots, and both the timescales of relaxation and the length-scales of signal propagation depend sensitively on the inputs as power laws, both diverging as the input to 0. The basic dynamical mechanism is that inputs to the network generate ongoing activity, which in turn controls how additional inputs or signals propagate spatially or attenuate in time. We present analytical solutions for the steady states when the network is forced with a single oscillation and when a background value creates a steady state of ongoing activity, and derive the relationships shaping the value of the temporal decay and spatial propagation length as a function of this background value.