Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks

TL;DR

This paper establishes new generalization bounds for homogeneous neural networks trained with stochastic gradient descent, showing that slower stepsize decay rates are possible, which better align with practical training scenarios.

Contribution

It proves that homogeneous neural networks allow for a slower stepsize decay of order 1/a0 extbackslash sqrt{t}a0, extending stability-based generalization bounds beyond previous constraints.

Findings

Slower stepsize decay 1/a0 extbackslash sqrt{t}a0 is sufficient for generalization.

Bounds are applicable to ReLU and LeakyReLU networks.

Theoretical extension to non-Lipschitz regimes.

Abstract

Algorithmic stability is among the most potent techniques in generalization analysis. However, its derivation usually requires a stepsize $\eta_t = \mathcal{O}(1/t)$ under non-convex training regimes, where $t$ denotes iterations. This rigid decay of the stepsize potentially impedes optimization and may not align with practical scenarios. In this paper, we derive the generalization bounds under the homogeneous neural network regimes, proving that this regime enables slower stepsize decay of order $\Omega(1/\sqrt{t})$ under mild assumptions. We further extend the theoretical results from several aspects, e.g., non-Lipschitz regimes. This finding is broadly applicable, as homogeneous neural networks encompass fully-connected and convolutional neural networks with ReLU and LeakyReLU activations.