Stochastic Modified Equations for Stochastic Gradient Descent in Infinite-Dimensional Hilbert Spaces

TL;DR

This paper extends diffusion approximation results for stochastic gradient descent (SGD) to infinite-dimensional Hilbert spaces, modeling the dynamics with stochastic differential equations driven by cylindrical Brownian motion.

Contribution

It develops a theoretical framework for analyzing the continuous-time limit of SGD in infinite-dimensional spaces, addressing challenges posed by cylindrical noise and weak convergence.

Findings

SGD dynamics can be approximated by a stochastic differential equation in Hilbert spaces.

The discrepancy between SGD and the SDE is second order in step size when evaluated with smooth functionals.

Numerical experiments confirm the convergence predictions.

Abstract

Inverse problems in scientific computing often require optimization over infinite-dimensional Hilbert spaces. A commonly used solver in such settings is stochastic gradient descent (SGD), where gradients are approximated using randomly sampled sub-objective functions. In this work we study the continuous-time limit of SGD in the small step-size regime. We show that the discrete dynamics can be approximated by a stochastic differential equation (SDE) driven by cylindrical Brownian motion. The analysis extends diffusion-approximation results previously established in Euclidean spaces to the infinite-dimensional setting. Two analytical difficulties arise in this extension. First, the cylindrical nature of the noise requires establishing well-posedness of the resulting stochastic evolution equation through appropriate structural conditions on the covariance operator. Second, since the randomness in SGD originates from discrete sampling while the limiting equation is driven by Gaussian noise, the comparison between the two dynamics must be carried out in a weak sense. We therefore introduce a suitable class of smooth functionals on the Hilbert space and prove that the discrepancy between SGD and the limiting SDE, when evaluated through these functionals, is of second order in the step size. Numerical experiments confirm the predicted convergence behavior.