Laplacian Smoothing Gradient Descent

TL;DR

This paper introduces a simple modification to gradient descent using Laplacian-based smoothing, which reduces variance, allows larger steps, and improves generalization across various machine learning models.

Contribution

It proposes a novel Laplacian smoothing technique for gradient descent that enhances convergence and generalization, supported by theoretical analysis and practical efficiency.

Findings

Reduces variance in gradient estimates

Enables larger step sizes in optimization

Improves generalization accuracy in neural networks

Abstract

We propose a class of very simple modifications of gradient descent and stochastic gradient descent. We show that when applied to a large variety of machine learning problems, ranging from logistic regression to deep neural nets, the proposed surrogates can dramatically reduce the variance, allow to take a larger step size, and improve the generalization accuracy. The methods only involve multiplying the usual (stochastic) gradient by the inverse of a positive definitive matrix (which can be computed efficiently by FFT) with a low condition number coming from a one-dimensional discrete Laplacian or its high order generalizations. It also preserves the mean and increases the smallest component and decreases the largest component. The theory of Hamilton-Jacobi partial differential equations demonstrates that the implicit version of the new algorithm is almost the same as doing gradient descent on a new function which (i) has the same global minima as the original function and (ii) is ``more convex". Moreover, we show that optimization algorithms with these surrogates converge uniformly in the discrete Sobolev $H_\sigma^p$ sense and reduce the optimality gap for convex optimization problems. The code is available at: \url{https://github.com/BaoWangMath/LaplacianSmoothing-GradientDescent}