Homeostatic Ubiquity of Hebbian Dynamics in Regularized Learning Rules

TL;DR

This paper reveals that regularized learning rules naturally develop Hebbian-like signals, providing a unified theoretical framework for understanding Hebbian and anti-Hebbian plasticity observed in biological brains.

Contribution

It demonstrates that L2 regularization induces Hebbian-like behavior in various learning rules and explains the emergence of anti-Hebbian plasticity from noise, unifying biological observations with theoretical models.

Findings

Regularized learning signals align with Hebbian signals at stationarity.

Anti-Hebbian plasticity can arise from gradient or input noise.

Most learning rules exhibit Hebbian-like behavior before convergence.

Abstract

Hebbian and anti-Hebbian plasticity are widely observed in the biological brain, yet their theoretical understanding remains limited. In this work, we find that when a learning method is regularized with L2 weight decay, its learning signal will gradually align with the direction of the Hebbian learning signal as it approaches stationarity. This Hebbian-like behavior is not unique to SGD: almost any learning rule, including random ones, can exhibit the same signature long before learning has ceased. We also provide a theoretical explanation for anti-Hebbian plasticity in regression tasks, demonstrating how it can arise naturally from gradient or input noise, and offering a potential reason for the observed anti-Hebbian effects in the brain. Certainly, our proposed mechanisms do not rule out any conventionally established forms of Hebbian plasticity and could coexist with them extensively in the brain. A key insight for neurophysiology is the need to develop ways to experimentally distinguish these two types of Hebbian observations.