Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learning

TL;DR

This paper explores how the Representer theorem can serve as a powerful framework for deriving biologically plausible neural learning algorithms, linking duality principles with Hebbian learning.

Contribution

It demonstrates that the Representer theorem provides a natural lens to understand and develop biologically plausible supervised learning algorithms using dual formulations.

Findings

The Representer theorem clarifies neural architecture emergence.

Dual formulations facilitate biologically plausible updates.

The approach connects regularized learning with Hebbian principles.

Abstract

A normative approach called Similarity Matching was recently introduced for deriving and understanding the algorithmic basis of neural computation focused on unsupervised problems. It involves deriving algorithms from computational objectives and evaluating their compatibility with anatomical and physiological observations. In particular, it introduces neural architectures by considering dual alternatives instead of primal formulations of popular models such as PCA. However, its connection to the Representer theorem remains unexplored. In this work, we propose to use teachings from this approach to explore supervised learning algorithms and clarify the notion of Hebbian learning. We examine regularized supervised learning and elucidate the emergence of neural architecture and additive versus multiplicative update rules. In this work, we focus not on developing new algorithms but on showing that the Representer theorem offers the perfect lens to study biologically plausible learning algorithms. We argue that many past and current advancements in the field rely on some form of dual formulation to introduce biological plausibility. In short, as long as a dual formulation exists, it is possible to derive biologically plausible algorithms. Our work sheds light on the pivotal role of the Representer theorem in advancing our comprehension of neural computation.