Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network

TL;DR

This paper investigates how stochastic gradient descent causes representational drift in a two-layer neural network, providing a theoretical framework that explains the dynamics and stimulus-dependent characteristics of the drift.

Contribution

It introduces a theoretical analysis of SGD-induced drift in a two-layer network, decomposing dynamics into normal and tangent components and deriving explicit fluctuation and diffusion coefficients.

Findings

Drift rate is slower for more frequently presented stimuli.

Analytical expressions for fluctuation and diffusion coefficients are derived.

The framework aligns with experimental observations of stimulus-dependent drift.

Abstract

Representational drift refers to over-time changes in neural activation accompanied by a stable task performance. Despite being observed in the brain and in artificial networks, the mechanisms of drift and its implications are not fully understood. Motivated by recent experimental findings of stimulus-dependent drift in the piriform cortex, we use theory and simulations to study this phenomenon in a two-layer linear feedforward network. Specifically, in a continual online learning scenario, we study the drift induced by the noise inherent in the Stochastic Gradient Descent (SGD). By decomposing the learning dynamics into the normal and tangent spaces of the minimum-loss manifold, we show the former corresponds to a finite variance fluctuation, while the latter could be considered as an effective diffusion process on the manifold. We analytically compute the fluctuation and the diffusion coefficients for the stimuli representations in the hidden layer as functions of network parameters and input distribution. Further, consistent with experiments, we show that the drift rate is slower for a more frequently presented stimulus. Overall, our analysis yields a theoretical framework for better understanding of the drift phenomenon in biological and artificial neural networks.