When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?

TL;DR

This paper analyzes when gradient descent on deep networks with smoothed ReLU activations drives the logistic loss to zero, providing convergence conditions and bounds applicable to various activation functions.

Contribution

It establishes convergence conditions for gradient descent on deep networks with smoothed ReLU activations, extending prior analyses to new activation functions.

Findings

Gradient descent drives logistic loss to zero under certain initial loss bounds.

Convergence is guaranteed under data separation conditions.

Applicable to smoothed ReLU variants like Swish and Huberized ReLU.

Abstract

We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero, and prove bounds on the rate of convergence. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU, proposed in previous applied work. We provide two sufficient conditions for convergence. The first is simply a bound on the loss at initialization. The second is a data separation condition used in prior analyses.