The effect of Leaky ReLUs on the training and generalization of overparameterized networks

TL;DR

This paper analyzes how leaky ReLU activation functions influence training and generalization errors in overparameterized neural networks, providing bounds that depend on the Leaky ReLU parameter and identifying the optimal value.

Contribution

It offers theoretical bounds on training and generalization errors for overparameterized networks with leaky ReLUs, highlighting the optimality of the absolute value activation.

Findings

α = -1 (absolute value) yields optimal training error bounds

Numerical experiments support the theoretical optimality of α = -1

Bounds depend on Leaky ReLU parameter, guiding practical choices

Abstract

We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $\alpha$. We show that $\alpha =-1$, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory.