Non-asymptotic convergence analysis of the stochastic gradient Hamiltonian Monte Carlo algorithm with discontinuous stochastic gradient with applications to training of ReLU neural networks

TL;DR

This paper analyzes the convergence of the stochastic gradient Hamiltonian Monte Carlo algorithm with discontinuous stochastic gradients, providing explicit bounds applicable to training neural networks with ReLU activations.

Contribution

It offers the first non-asymptotic convergence analysis of SGHMC with discontinuous gradients, applicable to non-convex optimization and neural network training.

Findings

Explicit upper bounds on expected excess risk are derived.

The analysis applies to neural networks with ReLU activation.

Numerical experiments demonstrate practical relevance.

Abstract

In this paper, we provide a non-asymptotic analysis of the convergence of the stochastic gradient Hamiltonian Monte Carlo (SGHMC) algorithm to a target measure in Wasserstein-1 and Wasserstein-2 distance. Crucially, compared to the existing literature on SGHMC, we allow its stochastic gradient to be discontinuous. This allows us to provide explicit upper bounds, which can be controlled to be arbitrarily small, for the expected excess risk of non-convex stochastic optimization problems with discontinuous stochastic gradients, including, among others, the training of neural networks with ReLU activation function. To illustrate the applicability of our main results, we consider numerical experiments on quantile estimation and on several optimization problems involving ReLU neural networks relevant in finance and artificial intelligence.