Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules

TL;DR

This paper introduces a theoretical framework that models learning rules as policies for navigating loss landscapes, unifying various algorithms and providing a foundation for designing adaptive learning methods.

Contribution

It presents a normative framework that derives learning rules as solutions to an optimal control problem, connecting many existing algorithms under a single theoretical perspective.

Findings

Gradient descent emerges from short-horizon optimization.

Momentum corresponds to longer-horizon planning.

Adaptive optimizers like Adam relate to Bayesian inference.

Abstract

Learning rules -- prescriptions for updating model parameters to improve performance -- are typically assumed rather than derived. Why do some learning rules work better than others, and under what assumptions can a given rule be considered optimal? We propose a theoretical framework that casts learning rules as policies for navigating (partially observable) loss landscapes, and identifies optimal rules as solutions to an associated optimal control problem. A range of well-known rules emerge naturally within this framework under different assumptions: gradient descent from short-horizon optimization, momentum from longer-horizon planning, natural gradients from accounting for parameter space geometry, non-gradient rules from partial controllability, and adaptive optimizers like Adam from online Bayesian inference of loss landscape shape. We further show that continual learning strategies like weight resetting can be understood as optimal responses to task uncertainty. By unifying these phenomena under a single objective, our framework clarifies the computational structure of learning and offers a principled foundation for designing adaptive algorithms.