Paradoxical noise preference in RNNs

TL;DR

Recurrent neural networks often perform best with a nonzero level of training noise due to noise-induced shifts in their fixed points, which can bias outputs when noise is removed, revealing a paradoxical noise preference.

Contribution

This study uncovers the counterintuitive phenomenon that RNNs trained with noise prefer nonzero noise levels at test time, linked to fixed point shifts in stochastic dynamics.

Findings

Networks trained with internal noise perform best at the same noise level.

Noise-induced fixed point shifts bias network outputs when noise is removed.

Performance degradation occurs when operating near activation nonlinearities.

Abstract

In recurrent neural networks (RNNs) used to model biological neural networks, noise is typically introduced during training to emulate biological variability and regularize learning. The expectation is that removing the noise at test time should preserve or improve performance. Contrary to this intuition, we find that continuous-time recurrent neural networks (CTRNNs) often perform best at a nonzero noise level, specifically, the same level used during training. This noise preference typically arises when noise is injected inside the neural activation function; networks trained with noise injected outside the activation function perform best with zero noise. Through analyses of simple function approximation, maze navigation, and single neuron regulator tasks, we show that the phenomenon stems from noise-induced shifts of fixed points (stationary distributions) in the underlying stochastic dynamics of the RNNs. These fixed point shifts are noise-level dependent and bias the network outputs when the noise is removed, degrading performance. Analytical and numerical results show that the bias arises when neural states operate near activation function nonlinearities, where noise is asymmetrically attenuated, and that performance optimization incentivizes operation near these nonlinearities. Thus, networks can overfit to the stochastic training environment itself rather than just to the input-output data. The phenomenon is distinct from stochastic resonance, wherein nonzero noise enhances signal processing. Our findings reveal that training noise can become an integral part of the computation learned by recurrent networks, with implications for understanding neural population dynamics and for the design of robust artificial RNNs.