Gradient descent in higher codimension

TL;DR

This paper investigates how noise influences gradient descent trajectories in complex settings with higher codimension minima, extending previous work from codimension 1 to more intricate cases through computer experiments.

Contribution

It explores the effects of noise on gradient descent in higher codimension minima, providing new insights beyond prior studies focused on codimension 1.

Findings

Noise affects gradient descent trajectories in higher codimension minima

Computer experiments reveal complex behaviors in these settings

Extends understanding of noisy gradient descent beyond simple minima

Abstract

We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had codimension 1. In this note, we do some computer experiments and observe the behavior of noisy gradient descent in the more complex setting of minima of higher codimension.