High-Dimensional Non-Convex Landscapes and Gradient Descent Dynamics

TL;DR

This paper explores how statistical physics methods can be applied to understand gradient descent behavior in complex, high-dimensional non-convex landscapes common in machine learning, providing new insights into optimization dynamics.

Contribution

It introduces physics-inspired approaches to analyze high-dimensional non-convex optimization landscapes in machine learning, bridging physics and AI research.

Findings

Application of statistical physics methods to gradient descent analysis

Insights into the dynamics of optimization in high-dimensional spaces

Framework for studying non-convex landscapes in machine learning

Abstract

In these lecture notes we present different methods and concepts developed in statistical physics to analyze gradient descent dynamics in high-dimensional non-convex landscapes. Our aim is to show how approaches developed in physics, mainly statistical physics of disordered systems, can be used to tackle open questions on high-dimensional dynamics in Machine Learning.